mijuanlo/nllb-200-distilled-1.3B-int8-ct2

NLLB-200 1.3B INT8 CTranslate2

Fast-Inference with CTranslate2

Speedup inference while reducing memory by 2x-4x using int8 quantization in C++ on CPU or GPU.

This is a quantized version of facebook/nllb-200-distilled-1.3B converted to CTranslate2 format.

pip install ctranslate2 sentencepiece

Checkpoint compatible with CTranslate2 >= 3.22.0

• compute_type=int8_float16 for device="cuda"
• compute_type=int8 for device="cpu"

Usage example

import ctranslate2
import sentencepiece as spm

model_path = "/path/to/nllb13"
spm_path = f"{model_path}/sentencepiece.bpe.model"

sp = spm.SentencePieceProcessor()
sp.load(spm_path)

translator = ctranslate2.Translator(model_path, device="cuda")

src_lang = "eng_Latn"
tgt_lang = "spa_Latn"

source = ["Hello, how are you?", "This is a test sentence."]
source_subworded = sp.encode_as_pieces(source)
source_subworded = [[src_lang] + pieces + ["</s>"] for pieces in source_subworded]
target_prefix = [[tgt_lang]] * len(source)

translations = translator.translate_batch(
    source_subworded,
    batch_type="tokens",
    max_batch_size=1024,
    beam_size=4,
    target_prefix=target_prefix,
)

for translation in translations:
    print(sp.decode(translation.hypotheses[0]))

Conversion command

ct2-transformers-converter --model /tmp/hf_model_dir --output_dir /opt/ai/traduia-ng/models/nllb13 --quantization int8 --force

License

This model is a quantized version of facebook/nllb-200-distilled-1.3B, which is released under the CC-BY-NC-4.0 license. The same license applies to this conversion.

Model description

NLLB-200 (No Language Left Behind) is a machine translation model from Meta AI that supports translation across 200 languages. This is the distilled 1.3B parameter variant, quantized to INT8 with CTranslate2.

• Paper: No Language Left Behind: Scaling Human-Centered Machine Translation (NLLB Team, 2022)
• Original model: facebook/nllb-200-distilled-1.3B
• Repository: github.com/facebookresearch/fairseq

Intended Use

Research in machine translation, especially for low-resource languages. Single sentence translation among 200 languages.

Metrics

Evaluated using BLEU, spBLEU, and chrF++ on the Flores-200 dataset.

Evaluation Data

Flores-200 dataset, preprocessed with SentencePiece.

Training Data

Parallel multilingual data from multiple sources and monolingual data from Common Crawl. See Section 5 of the paper for details.

Ethical Considerations

The original NLLB-200 model may produce mistranslations that could have adverse impacts if used for critical decisions (health, safety). Training data mined from public web sources may contain personally identifiable information despite cleaning efforts. This model should not be used for certified translations or domain-specific text (medical, legal).

Caveats and Recommendations

Tested primarily on Wikimedia domain. Input length should not exceed 512 tokens. Language variations may not be fully captured. Users should make appropriate assessments for their use case.

Repository

Hugging Face: mijuanlo/nllb-200-distilled-1.3B-int8-ct2