File size: 1,996 Bytes
7a4e276 ecc8463 7a4e276 ecc8463 7a4e276 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 | ---
library_name: pytorch
pipeline_tag: translation
tags:
- arabic
- english
- machine-translation
---
# From Subwords to Pixels: Evaluating Text Representations for Arabic Machine Translation
Final checkpoints for the Base, Vocab+, Byte, and Pixel Arabic-to-English
machine translation models described in our paper on [From Subwords to Pixels: Evaluating Text Representations for Arabic Machine Translation]().
These models are intended to be used with this repo: [mbzuai-nlp/arapixel-mt](https://github.com/mbzuai-nlp/arapixel-mt).
## Layout
Each model has MSA-pretrained checkpoints for 5%, 10%, 25%, 50%, and the full
MSA training set:
```text
<model>/msa/{5pct,10pct,25pct,50pct,full}/pytorch_model.bin
```
The curriculum checkpoints were fine-tuned on the 1:1 MSA/dialect mixture:
```text
<model>/finetuned/{5pct,10pct,25pct,50pct}/msa_dialect_1x/best_model.bin
```
Full-data checkpoints have four fine-tuning variants:
```text
<model>/finetuned/full/dialect/best_model.bin
<model>/finetuned/full/msa_dialect_1x/best_model.bin
<model>/finetuned/full/msa_dialect_2x/best_model.bin
<model>/finetuned/full/msa_dialect_4x/best_model.bin
```
`<model>` is one of `pixel`, `bytes`, `vocab`, or `base`. In the codebase,
the Base model is selected with `--model decoder`.
Every checkpoint directory also contains `training_complete.json`, which lets
evaluation select the fixed MSA or dynamic fine-tuning source layout. The shared
Vocab+ tokenizer is stored in `vocab/tokenizer/`.
## Download one checkpoint
```bash
hf download shaheen6/PixelTranslationCheckpoints --include "pixel/msa/full/*" --local-dir checkpoints
```
The downloaded model path is then:
```text
checkpoints/pixel/msa/full/pytorch_model.bin
```
Pass that file to `--model-checkpoint`. For Vocab+, also download the tokenizer:
```bash
hf download shaheen6/PixelTranslationCheckpoints --include "vocab/tokenizer/*" --local-dir checkpoints
export VOCAB_TOKENIZER_DIR="$PWD/checkpoints/vocab/tokenizer"
```
|