diff --git a/data/fairseq/examples/audio_nlp/nlu/README.md b/data/fairseq/examples/audio_nlp/nlu/README.md new file mode 100644 index 0000000000000000000000000000000000000000..a11b3f306584c81a74aa7bf768d5fd264f177cd9 --- /dev/null +++ b/data/fairseq/examples/audio_nlp/nlu/README.md @@ -0,0 +1,53 @@ +# End-to-end NLU + +End-to-end spoken language understanding (SLU) predicts intent directly from audio using a single model. It promises to improve the performance of assistant systems by leveraging acoustic information lost in the intermediate textual representation and preventing cascading errors from Automatic Speech Recognition (ASR). Further, having one unified model has efficiency advantages when deploying assistant systems on-device. + +This page releases the code for reproducing the results in [STOP: A dataset for Spoken Task Oriented Semantic Parsing](https://arxiv.org/abs/2207.10643) + +The dataset can be downloaded here: [download link](https://dl.fbaipublicfiles.com/stop/stop.tar.gz) + +The low-resource splits can be downloaded here: [download link](http://dl.fbaipublicfiles.com/stop/low_resource_splits.tar.gz) + +## Pretrained models end-to-end NLU Models + +| Speech Pretraining | ASR Pretraining | Test EM Accuracy | Tesst EM-Tree Accuracy | Link | +| ----------- | ----------- |----------|----------|----------| +| None | None | 36.54 | 57.01 | [link](https://dl.fbaipublicfiles.com/stop/end-to-end-nlu-none-none.pt) | +| Wav2Vec | None | 68.05 | 82.53 | [link](https://dl.fbaipublicfiles.com/stop/end-to-end-nlu-wav2vec-none.pt) | +| HuBERT | None | 68.40 | 82.85 | [link](https://dl.fbaipublicfiles.com/stop/end-to-end-nlu-hubert-none.pt) | +| Wav2Vec | STOP | 68.70 | 82.78 | [link](https://dl.fbaipublicfiles.com/stop/end-to-end-nlu-wav2vec-stop.pt) | +| HuBERT | STOP | 69.23 | 82.87 | [link](https://dl.fbaipublicfiles.com/stop/end-to-end-nlu-hubert-stop.pt) | +| Wav2Vec | Librispeech | 68.47 | 82.49 | [link](https://dl.fbaipublicfiles.com/stop/end-to-end-nlu-wav2vec-ls.pt) | +| HuBERT | Librispeech | 68.70 | 82.78 | [link](https://dl.fbaipublicfiles.com/stop/end-to-end-nlu-hubert-ls.pt) | + +## Pretrained models ASR Models +| Speech Pre-training | ASR Dataset | STOP Eval WER | STOP Test WER | dev\_other WER | dev\_clean WER | test\_clean WER | test\_other WER | Link | +| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | +| HuBERT | Librispeech | 8.47 | 2.99 | 3.25 | 8.06 | 25.68 | 26.19 | [link](https://dl.fbaipublicfiles.com/stop/ctc-asr-hubert-ls.pt) | +| Wav2Vec | Librispeech | 9.215 | 3.204 | 3.334 | 9.006 | 27.257 | 27.588 | [link](https://dl.fbaipublicfiles.com/stop/ctc-asr-wav2vec-ls.pt) | +| HuBERT | STOP | 46.31 | 31.30 | 31.52 | 47.16 | 4.29 | 4.26 | [link](https://dl.fbaipublicfiles.com/stop/ctc-asr-hubert-stop.pt) | +| Wav2Vec | STOP | 43.103 | 27.833 | 28.479 | 28.479 | 4.679 | 4.667 | [link](https://dl.fbaipublicfiles.com/stop/ctc-asr-wav2vec-stop.pt) | +| HuBERT | Librispeech + STOP | 9.015 | 3.211 | 3.372 | 8.635 | 5.133 | 5.056 | [link](https://dl.fbaipublicfiles.com/stop/ctc-asr-hubert-ls-stop.pt) | +| Wav2Vec | Librispeech + STOP | 9.549 | 3.537 | 3.625 | 9.514 | 5.59 | 5.562 | [link](https://dl.fbaipublicfiles.com/stop/ctc-asr-wav2vec-ls-stop.pt) | + +## Creating the fairseq datasets from STOP + +First, create the audio file manifests and label files: + +``` +python examples/audio_nlp/nlu/generate_manifests.py --stop_root $STOP_DOWNLOAD_DIR/stop --output $FAIRSEQ_DATASET_OUTPUT/ +``` + + +Run `./examples/audio_nlp/nlu/create_dict_stop.sh $FAIRSEQ_DATASET_OUTPUT` to generate the fairseq dictionaries. + + +## Training an End-to-end NLU Model + + +Download a wav2vec or hubert model from [link](https://github.com/facebookresearch/fairseq/tree/main/examples/hubert) or [link](https://github.com/facebookresearch/fairseq/tree/main/examples/wav2vec) + + +``` +python fairseq_cli/hydra-train --config-dir examples/audio_nlp/nlu/configs/ --config-name nlu_finetuning task.data=$FAIRSEQ_DATA_OUTPUT model.w2v_path=$PRETRAINED_MODEL_PATH +``` diff --git a/data/fairseq/examples/audio_nlp/nlu/configs/nlu_finetuning.yaml b/data/fairseq/examples/audio_nlp/nlu/configs/nlu_finetuning.yaml new file mode 100644 index 0000000000000000000000000000000000000000..bb90f45a307bd36040ba579a012bac8db911ec5c --- /dev/null +++ b/data/fairseq/examples/audio_nlp/nlu/configs/nlu_finetuning.yaml @@ -0,0 +1,59 @@ +# @package _group_ + +common: + fp16: true + log_format: json + log_interval: 10 + tensorboard_logdir: tb + +checkpoint: + no_epoch_checkpoints: true + best_checkpoint_metric: em_error + save_interval: 10 + +task: + _name: nlu_finetuning + data: ??? + labels: parse + eval_wer_parse: true + autoregressive: true + +dataset: + num_workers: 6 + max_tokens: 1600000 + skip_invalid_size_inputs_valid_test: true + valid_subset: eval,test + train_subset: train + validate_interval: 10 + +criterion: + _name: label_smoothed_cross_entropy + +optimization: + max_update: 320000 + lr: [0.0001] + sentence_avg: true + update_freq: [1] + +optimizer: + _name: adam + adam_betas: (0.9,0.98) + adam_eps: 1e-08 + +lr_scheduler: + _name: tri_stage + phase_ratio: [0.1, 0.4, 0.5] + final_lr_scale: 0.05 + +model: + _name: wav2vec_seq2seq + w2v_path: ??? + autoregressive: true + apply_mask: true + mask_prob: 0.5 + mask_channel_prob: 0.5 + mask_channel_length: 64 + layerdrop: 0.1 + activation_dropout: 0.1 + feature_grad_mult: 0.0 + freeze_finetune_updates: 0 diff --git a/data/fairseq/examples/audio_nlp/nlu/create_dict_stop.sh b/data/fairseq/examples/audio_nlp/nlu/create_dict_stop.sh new file mode 100644 index 0000000000000000000000000000000000000000..753393284de3703247db31ab224ebf11fab0242b --- /dev/null +++ b/data/fairseq/examples/audio_nlp/nlu/create_dict_stop.sh @@ -0,0 +1,38 @@ +#!/bin/bash + +### Script handling creation of data binaries +### for model training within fairseq + + +fairseq_root="." + +data_root=$1 +train_prefix="${data_root}/train" +valid_prefix="${data_root}/eval" +test_prefix="${data_root}/test" + +dest_dir="$data_root/" + +#echo "src dict: $src_dict" > "$dest_dir/src_dict.txt" +#echo "trg dict: $tgt_dict" > "$dest_dir/tgt_dict.txt" + + #--tgtdict $tgt_dict \ +PYTHONPATH=$fairseq_root \ + python $fairseq_root/fairseq_cli/preprocess.py \ + --source-lang "parse" \ + --trainpref "$train_prefix" \ + --validpref "$valid_prefix" \ + --destdir "$dest_dir" \ + --only-source \ + --dict-only \ + --workers 60; + +PYTHONPATH=$fairseq_root \ + python $fairseq_root/fairseq_cli/preprocess.py \ + --source-lang "ltr" \ + --trainpref "$train_prefix" \ + --validpref "$valid_prefix" \ + --destdir "$dest_dir" \ + --only-source \ + --dict-only \ + --workers 60; diff --git a/data/fairseq/examples/audio_nlp/nlu/generate_manifests.py b/data/fairseq/examples/audio_nlp/nlu/generate_manifests.py new file mode 100644 index 0000000000000000000000000000000000000000..e2176099cbd3993d3488ccc85b60f5a0da45d4df --- /dev/null +++ b/data/fairseq/examples/audio_nlp/nlu/generate_manifests.py @@ -0,0 +1,83 @@ +import argparse +from pathlib import Path +import soundfile + +def get_insl_frame(parse): + out = [] + def is_ont_token(tok): + return tok[0] in ["[", "]"]; + + res = [] + x = [] + for tok in parse.split(): + if is_ont_token(tok): + res.extend('_'.join(x)) + x = [] + res.append(tok.upper()) + else: + x.append(tok.upper()) + + return " ".join(res) + ' | ' + +def sequencify_utterance(utterance): + utterance = utterance.upper() + utterance = utterance.replace(' ', '|') + '|' + utterance = list(utterance) + utterance = ' '.join(utterance) + return utterance + + +def generate_fairseq_manifests(manifest, output_path, audio_root=None): + + with open(manifest, 'r') as i: + parses = [] + utterances = [] + filepaths = [] + keys = None + for (idx, line) in enumerate(i): + if idx == 0: keys = line.strip().split('\t') + else: + data = { k: v for (k, v) in zip(keys, line.split('\t'))} + parses.append(get_insl_frame(data['decoupled_normalized_seqlogical'])) + utterances.append(sequencify_utterance(data['normalized_utterance'])) + filepaths.append(data['file_id']) + + parses_fp = output_path.with_suffix('.parse') + with open(str(parses_fp), 'w') as o: + for p in parses: + o.write(p + '\n') + + utterances_fp = output_path.with_suffix('.ltr') + with open(str(utterances_fp), 'w') as o: + for u in utterances: + o.write(u + '\n') + + filepaths_fp = output_path.with_suffix('.tsv') + with open(str(filepaths_fp), 'w') as o: + o.write(str(audio_root) + '\n') + for f in filepaths: + fullpath = audio_root / f + assert fullpath.exists(), f'{fullpath}' + frames = soundfile.info(fullpath).frames + o.write(f'{f}\t{frames}\n') + +def main(args): + + splits = ['train', 'eval', 'test'] + root = Path(args.stop_root) + output_root = Path(args.output) + + for split in splits: + stop_manifest_path = root / 'manifests' / (split + '.tsv') + output_path = output_root / (split) + + generate_fairseq_manifests(stop_manifest_path, output_path, root) + +if __name__ == '__main__': + parser = argparse.ArgumentParser(description='Process some integers.') + parser.add_argument('--stop_root', type=str, + help='path to stop root directory') + parser.add_argument('--output', type=str, + help='output directory') + args = parser.parse_args() + main(args) diff --git a/data/fairseq/examples/emotion_conversion/README.md b/data/fairseq/examples/emotion_conversion/README.md new file mode 100644 index 0000000000000000000000000000000000000000..caf22befe4cfb6fa07d0f7e9c1268eb6ec9fc392 --- /dev/null +++ b/data/fairseq/examples/emotion_conversion/README.md @@ -0,0 +1,214 @@ +# Textless speech emotion conversion using decomposed and discrete representations +[Felix Kreuk](https://felixkreuk.github.io), Adam Polyak, Jade Copet, Eugene Kharitonov, Tu-Anh Nguyen, Morgane Rivière, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, [Yossi Adi](https://adiyoss.github.io) + +_abstract_: Speech emotion conversion is the task of modifying the perceived emotion of a speech utterance while preserving the lexical content and speaker identity. In this study, we cast the problem of emotion conversion as a spoken language translation task. We decompose speech into discrete and disentangled learned representations, consisting of content units, F0, speaker, and emotion. First, we modify the speech content by translating the content units to a target emotion, and then predict the prosodic features based on these units. Finally, the speech waveform is generated by feeding the predicted representations into a neural vocoder. Such a paradigm allows us to go beyond spectral and parametric changes of the signal, and model non-verbal vocalizations, such as laughter insertion, yawning removal, etc. We demonstrate objectively and subjectively that the proposed method is superior to the baselines in terms of perceived emotion and audio quality. We rigorously evaluate all components of such a complex system and conclude with an extensive model analysis and ablation study to better emphasize the architectural choices, strengths and weaknesses of the proposed method. Samples and code will be publicly available under the following link: https://speechbot.github.io/emotion. + +## Installation +First, create a conda virtual environment and activate it: +``` +conda create -n emotion python=3.8 -y +conda activate emotion +``` + +Then, clone this repository: +``` +git clone https://github.com/facebookresearch/fairseq.git +cd fairseq/examples/emotion_conversion +git clone https://github.com/felixkreuk/speech-resynthesis +``` + +Next, download the EmoV discrete tokens: +``` +wget https://dl.fbaipublicfiles.com/textless_nlp/emotion_conversion/data.tar.gz # (still in fairseq/examples/emotion_conversion) +tar -xzvf data.tar.gz +``` + +Your `fairseq/examples/emotion_conversion` directory should like this: +``` +drwxrwxr-x 3 felixkreuk felixkreuk 0 Feb 6 2022 data +drwxrwxr-x 3 felixkreuk felixkreuk 0 Sep 28 10:41 emotion_models +drwxr-xr-x 3 felixkreuk felixkreuk 0 Jun 29 05:43 fairseq_models +drwxr-xr-x 3 felixkreuk felixkreuk 0 Sep 28 10:41 preprocess +-rw-rw-r-- 1 felixkreuk felixkreuk 11K Dec 5 09:00 README.md +-rw-rw-r-- 1 felixkreuk felixkreuk 88 Mar 6 2022 requirements.txt +-rw-rw-r-- 1 felixkreuk felixkreuk 13K Jun 29 06:26 synthesize.py +``` + +Lastly, install fairseq and the other packages: +``` +pip install --editable ./ +pip install -r examples/emotion_conversion/requirements.txt +``` + +## Data preprocessing + +### Convert your audio to discrete representations +Please follow the steps described [here](https://github.com/pytorch/fairseq/tree/main/examples/hubert/simple_kmeans). +To generate the same discrete representations please use the following: +1. [HuBERT checkpoint](https://dl.fbaipublicfiles.com/hubert/hubert_base_ls960.pt) +2. k-means model at `data/hubert_base_ls960_layer9_clusters200/data_hubert_base_ls960_layer9_clusters200.bin` + +### Construct data splits +This step will use the discrete representations from the previous step and split them to train/valid/test sets for 3 tasks: +1. Translation model pre-training (BART language denoising) +2. Translation model training (content units emotion translation mechanism) +3. HiFiGAN model training (for synthesizing audio from discrete representations) + +Your processed data should be at `data/`: +1. `hubert_base_ls960_layer9_clusters200` - discrete representations extracted using HuBERT layer 9, clustered into 200 clusters. +2. `data.tsv` - a tsv file pointing to the EmoV dataset in your environment (Please edit the first line of this file according to your path). + +The following command will create the above splits: +``` +python examples/emotion_conversion/preprocess/create_core_manifest.py \ + --tsv data/data.tsv \ + --emov-km data/hubert_base_ls960_layer9_clusters200/data.km \ + --km data/hubert_base_ls960_layer9_clusters200/vctk.km \ + --dict data/hubert_base_ls960_layer9_clusters200/dict.txt \ + --manifests-dir $DATA +``` +* Set `$DATA` as the directory that will contain the processed data. + +### Extract F0 +To train the HiFiGAN vocoder we need to first extract the F0 curves: +``` +python examples/emotion_conversion/preprocess/extract_f0.py \ + --tsv data/data.tsv \ + --extractor pyaapt \ +``` + +## HiFiGAN training +Now we are all set to train the HiFiGAN vocoder: +``` +python examples/emotion_conversion/speech-resynthesis/train.py + --checkpoint_path \ + --config examples/emotion_conversion/speech-resynthesis/configs/EmoV/emov_hubert-layer9-cluster200_fixed-spkr-embedder_f0-raw_gst.json +``` + +## Translation Pre-training +Before translating emotions, we first need to pre-train the translation model as a denoising autoencoder (similarly to BART). +``` +python train.py \ + $DATA/fairseq-data/emov_multilingual_denoising_cross-speaker_dedup_nonzeroshot/tokenized \ + --save-dir \ + --tensorboard-logdir \ + --langs neutral,amused,angry,sleepy,disgusted,vctk.km \ + --dataset-impl mmap \ + --task multilingual_denoising \ + --arch transformer_small --criterion cross_entropy \ + --multilang-sampling-alpha 1.0 --sample-break-mode eos --max-tokens 16384 \ + --update-freq 1 --max-update 3000000 \ + --dropout 0.1 --attention-dropout 0.1 --relu-dropout 0.0 \ + --optimizer adam --weight-decay 0.01 --adam-eps 1e-06 \ + --clip-norm 0.1 --lr-scheduler polynomial_decay --lr 0.0003 \ + --total-num-update 3000000 --warmup-updates 10000 --fp16 \ + --poisson-lambda 3.5 --mask 0.3 --mask-length span-poisson --replace-length 1 --rotate 0 --mask-random 0.1 --insert 0 --permute-sentences 1.0 \ + --skip-invalid-size-inputs-valid-test \ + --user-dir examples/emotion_conversion/fairseq_models +``` + +## Translation Training +Now we are ready to train our emotion translation model: +``` +python train.py \ + --distributed-world-size 1 \ + $DATA/fairseq-data/emov_multilingual_translation_cross-speaker_dedup/tokenized/ \ + --save-dir \ + --tensorboard-logdir \ + --arch multilingual_small --task multilingual_translation \ + --criterion label_smoothed_cross_entropy --label-smoothing 0.2 \ + --lang-pairs neutral-amused,neutral-sleepy,neutral-disgusted,neutral-angry,amused-sleepy,amused-disgusted,amused-neutral,amused-angry,angry-amused,angry-sleepy,angry-disgusted,angry-neutral,disgusted-amused,disgusted-sleepy,disgusted-neutral,disgusted-angry,sleepy-amused,sleepy-neutral,sleepy-disgusted,sleepy-angry \ + --optimizer adam --adam-betas "(0.9, 0.98)" --adam-eps 1e-06 \ + --lr 1e-05 --clip-norm 0 --dropout 0.1 --attention-dropout 0.1 \ + --weight-decay 0.01 --warmup-updates 2000 --lr-scheduler inverse_sqrt \ + --max-tokens 4096 --update-freq 1 --max-update 100000 \ + --required-batch-size-multiple 8 --fp16 --num-workers 4 \ + --seed 2 --log-format json --log-interval 25 --save-interval-updates 1000 \ + --no-epoch-checkpoints --keep-best-checkpoints 1 --keep-interval-updates 1 \ + --finetune-from-model \ + --user-dir examples/emotion_conversion/fairseq_models +``` +* To share encoders/decoders use the `--share-encoders` and `--share-decoders` flags. +* To add source/target emotion tokens use the `--encoder-langtok {'src'|'tgt'}` and `--decoder-langtok` flags. + +## F0-predictor Training +The following command trains the F0 prediction module: +``` +cd examples/emotion_conversion +python -m emotion_models.pitch_predictor n_tokens=200 \ + train_tsv="$DATA/denoising/emov/train.tsv" \ + train_km="$DATA/denoising/emov/train.km" \ + valid_tsv="$DATA/denoising/emov/valid.tsv" \ + valid_km="$DATA/denoising/emov/valid.km" +``` +* See `hyra.run.dir` to configure directory for saving models. + +## Duration-predictor Training +The following command trains the duration prediction modules: +``` +cd examples/emotion_conversion +for emotion in "neutral" "amused" "angry" "disgusted" "sleepy"; do + python -m emotion_models.duration_predictor n_tokens=200 substring=$emotion \ + train_tsv="$DATA/denoising/emov/train.tsv" \ + train_km="$DATA/denoising/emov/train.km" \ + valid_tsv="$DATA/denoising/emov/valid.tsv" \ + valid_km="$DATA/denoising/emov/valid.km" +done +``` +* See `hyra.run.dir` to configure directory for saving models. +* After the above command you should have 5 duration models in your checkpoint directory: +``` +❯ ll duration_predictor/ +total 21M +-rw-rw-r-- 1 felixkreuk felixkreuk 4.1M Nov 15 2021 amused.ckpt +-rw-rw-r-- 1 felixkreuk felixkreuk 4.1M Nov 15 2021 angry.ckpt +-rw-rw-r-- 1 felixkreuk felixkreuk 4.1M Nov 15 2021 disgusted.ckpt +-rw-rw-r-- 1 felixkreuk felixkreuk 4.1M Nov 15 2021 neutral.ckpt +-rw-rw-r-- 1 felixkreuk felixkreuk 4.1M Nov 15 2021 sleepy.ckpt +``` + +## Token Generation +The following command uses `fairseq-generate` to generate the token sequences based on the source and target emotions. +``` +fairseq-generate \ + $DATA/fairseq-data/emov_multilingual_translation_cross-speaker_dedup/tokenized/ \ + --task multilingual_translation \ + --gen-subset test \ + --path \ + --beam 5 \ + --batch-size 4 --max-len-a 1.8 --max-len-b 10 --lenpen 1 --min-len 1 \ + --skip-invalid-size-inputs-valid-test --distributed-world-size 1 \ + --source-lang neutral --target-lang amused \ + --lang-pairs neutral-amused,neutral-sleepy,neutral-disgusted,neutral-angry,amused-sleepy,amused-disgusted,amused-neutral,amused-angry,angry-amused,angry-sleepy,angry-disgusted,angry-neutral,disgusted-amused,disgusted-sleepy,disgusted-neutral,disgusted-angry,sleepy-amused,sleepy-neutral,sleepy-disgusted,sleepy-angry \ + --results-path \ + --user-dir examples/emotion_conversion/fairseq_models +``` +* Modify `--source-lang` and `--target-lang` to control for the source and target emotions. +* See [fairseq documentation](https://fairseq.readthedocs.io/en/latest/command_line_tools.html#fairseq-generate) for a full overview of generation parameters (e.g., top-k/top-p sampling). + +## Waveform Synthesis +Using the output of the above command, the HiFiGAN vocoder, and the prosody prediction modules (F0 and duration) we can now generate the output waveforms: +``` +python examples/emotion_conversion/synthesize.py \ + --result-path /generate-test.txt \ + --data $DATA/fairseq-data/emov_multilingual_translation_cross-speaker_dedup/neutral-amused \ + --orig-tsv examples/emotion_conversion/data/data.tsv \ + --orig-km examples/emotion_conversion/data/hubert_base_ls960_layer9_clusters200/data.km \ + --checkpoint-file /g_00400000 \ + --dur-model duration_predictor/ \ + --f0-model pitch_predictor/pitch_predictor.ckpt \ + -s neutral -t amused \ + --outdir ~/tmp/emotion_results/wavs/neutral-amused +``` +* Please make sure the source and target emotions here match those of the previous command. + +# Citation +If you find this useful in your research, please use the following BibTeX entry for citation. +``` +@article{kreuk2021textless, + title={Textless speech emotion conversion using decomposed and discrete representations}, + author={Kreuk, Felix and Polyak, Adam and Copet, Jade and Kharitonov, Eugene and Nguyen, Tu-Anh and Rivi{\`e}re, Morgane and Hsu, Wei-Ning and Mohamed, Abdelrahman and Dupoux, Emmanuel and Adi, Yossi}, + journal={Conference on Empirical Methods in Natural Language Processing (EMNLP)}, + year={2022} +} +``` diff --git a/data/fairseq/examples/emotion_conversion/emotion_models/__init__.py b/data/fairseq/examples/emotion_conversion/emotion_models/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/data/fairseq/examples/emotion_conversion/emotion_models/duration_predictor.py b/data/fairseq/examples/emotion_conversion/emotion_models/duration_predictor.py new file mode 100644 index 0000000000000000000000000000000000000000..eb47df0a214b593987611e2d62c61c48a0ea0c67 --- /dev/null +++ b/data/fairseq/examples/emotion_conversion/emotion_models/duration_predictor.py @@ -0,0 +1,243 @@ +import logging +import os + +import hydra +import torch +import torch.nn as nn +import torch.nn.functional as F +from einops.layers.torch import Rearrange +from torch.utils.data import DataLoader, Dataset + +from .utils import Accuracy + +logger = logging.getLogger(__name__) + + +def save_ckpt(model, path, model_class): + ckpt = { + "state_dict": model.state_dict(), + "padding_token": model.padding_token, + "model_class": model_class, + } + torch.save(ckpt, path) + + +def load_ckpt(path): + ckpt = torch.load(path) + ckpt["model_class"]["_target_"] = "emotion_models.duration_predictor.CnnPredictor" + model = hydra.utils.instantiate(ckpt["model_class"]) + model.load_state_dict(ckpt["state_dict"]) + model.padding_token = ckpt["padding_token"] + model = model.cpu() + model.eval() + return model + + +class Collator: + def __init__(self, padding_idx): + self.padding_idx = padding_idx + + def __call__(self, batch): + x = [item[0] for item in batch] + lengths = [len(item) for item in x] + x = torch.nn.utils.rnn.pad_sequence(x, batch_first=True, padding_value=self.padding_idx) + y = [item[1] for item in batch] + y = torch.nn.utils.rnn.pad_sequence(y, batch_first=True, padding_value=self.padding_idx) + mask = (x != self.padding_idx) + return x, y, mask, lengths + + +class Predictor(nn.Module): + def __init__(self, n_tokens, emb_dim): + super(Predictor, self).__init__() + self.n_tokens = n_tokens + self.emb_dim = emb_dim + self.padding_token = n_tokens + # add 1 extra embedding for padding token, set the padding index to be the last token + # (tokens from the clustering start at index 0) + self.emb = nn.Embedding(n_tokens + 1, emb_dim, padding_idx=self.padding_token) + + def inflate_input(self, batch): + """ get a sequence of tokens, predict their durations + and inflate them accordingly """ + batch_durs = self.forward(batch) + batch_durs = torch.exp(batch_durs) - 1 + batch_durs = batch_durs.round() + output = [] + for seq, durs in zip(batch, batch_durs): + inflated_seq = [] + for token, n in zip(seq, durs): + if token == self.padding_token: + break + n = int(n.item()) + token = int(token.item()) + inflated_seq.extend([token for _ in range(n)]) + output.append(inflated_seq) + output = torch.LongTensor(output) + return output + + +class CnnPredictor(Predictor): + def __init__(self, n_tokens, emb_dim, channels, kernel, output_dim, dropout, n_layers): + super(CnnPredictor, self).__init__(n_tokens=n_tokens, emb_dim=emb_dim) + layers = [ + Rearrange("b t c -> b c t"), + nn.Conv1d(emb_dim, channels, kernel_size=kernel, padding=(kernel - 1) // 2), + Rearrange("b c t -> b t c"), + nn.ReLU(), + nn.LayerNorm(channels), + nn.Dropout(dropout), + ] + for _ in range(n_layers-1): + layers += [ + Rearrange("b t c -> b c t"), + nn.Conv1d(channels, channels, kernel_size=kernel, padding=(kernel - 1) // 2), + Rearrange("b c t -> b t c"), + nn.ReLU(), + nn.LayerNorm(channels), + nn.Dropout(dropout), + ] + self.conv_layer = nn.Sequential(*layers) + self.proj = nn.Linear(channels, output_dim) + + def forward(self, x): + x = self.emb(x) + x = self.conv_layer(x) + x = self.proj(x) + x = x.squeeze(-1) + return x + + +def l2_log_loss(input, target): + return F.mse_loss( + input=input.float(), + target=torch.log(target.float() + 1), + reduce=False + ) + + +class DurationDataset(Dataset): + def __init__(self, tsv_path, km_path, substring=""): + lines = open(tsv_path, "r").readlines() + self.root, self.tsv = lines[0], lines[1:] + self.km = open(km_path, "r").readlines() + logger.info(f"loaded {len(self.km)} files") + + if substring != "": + tsv, km = [], [] + for tsv_line, km_line in zip(self.tsv, self.km): + if substring.lower() in tsv_line.lower(): + tsv.append(tsv_line) + km.append(km_line) + self.tsv, self.km = tsv, km + logger.info(f"after filtering: {len(self.km)} files") + + def __len__(self): + return len(self.km) + + def __getitem__(self, i): + x = self.km[i] + x = x.split(" ") + x = list(map(int, x)) + + y = [] + xd = [] + count = 1 + for x1, x2 in zip(x[:-1], x[1:]): + if x1 == x2: + count += 1 + continue + else: + y.append(count) + xd.append(x1) + count = 1 + + xd = torch.LongTensor(xd) + y = torch.LongTensor(y) + return xd, y + + +def train(cfg): + device = "cuda:0" + model = hydra.utils.instantiate(cfg[cfg.model]).to(device) + optimizer = hydra.utils.instantiate(cfg.optimizer, model.parameters()) + # add 1 extra embedding for padding token, set the padding index to be the last token + # (tokens from the clustering start at index 0) + collate_fn = Collator(padding_idx=model.padding_token) + logger.info(f"data: {cfg.train_tsv}") + train_ds = DurationDataset(cfg.train_tsv, cfg.train_km, substring=cfg.substring) + valid_ds = DurationDataset(cfg.valid_tsv, cfg.valid_km, substring=cfg.substring) + train_dl = DataLoader(train_ds, batch_size=32, shuffle=True, collate_fn=collate_fn) + valid_dl = DataLoader(valid_ds, batch_size=32, shuffle=False, collate_fn=collate_fn) + + best_loss = float("inf") + for epoch in range(cfg.epochs): + train_loss, train_loss_scaled = train_epoch(model, train_dl, l2_log_loss, optimizer, device) + valid_loss, valid_loss_scaled, *acc = valid_epoch(model, valid_dl, l2_log_loss, device) + acc0, acc1, acc2, acc3 = acc + if valid_loss_scaled < best_loss: + path = f"{os.getcwd()}/{cfg.substring}.ckpt" + save_ckpt(model, path, cfg[cfg.model]) + best_loss = valid_loss_scaled + logger.info(f"saved checkpoint: {path}") + logger.info(f"[epoch {epoch}] train loss: {train_loss:.3f}, train scaled: {train_loss_scaled:.3f}") + logger.info(f"[epoch {epoch}] valid loss: {valid_loss:.3f}, valid scaled: {valid_loss_scaled:.3f}") + logger.info(f"acc: {acc0,acc1,acc2,acc3}") + + +def train_epoch(model, loader, criterion, optimizer, device): + model.train() + epoch_loss = 0 + epoch_loss_scaled = 0 + for x, y, mask, _ in loader: + x, y, mask = x.to(device), y.to(device), mask.to(device) + yhat = model(x) + loss = criterion(yhat, y) * mask + loss = torch.mean(loss) + loss.backward() + nn.utils.clip_grad_norm_(model.parameters(), 1.0) + optimizer.step() + epoch_loss += loss.item() + # get normal scale loss + yhat_scaled = torch.exp(yhat) - 1 + yhat_scaled = torch.round(yhat_scaled) + scaled_loss = torch.mean(torch.abs(yhat_scaled - y) * mask) + epoch_loss_scaled += scaled_loss.item() + return epoch_loss / len(loader), epoch_loss_scaled / len(loader) + + +def valid_epoch(model, loader, criterion, device): + model.eval() + epoch_loss = 0 + epoch_loss_scaled = 0 + acc = Accuracy() + for x, y, mask, _ in loader: + x, y, mask = x.to(device), y.to(device), mask.to(device) + yhat = model(x) + loss = criterion(yhat, y) * mask + loss = torch.mean(loss) + epoch_loss += loss.item() + # get normal scale loss + yhat_scaled = torch.exp(yhat) - 1 + yhat_scaled = torch.round(yhat_scaled) + scaled_loss = torch.sum(torch.abs(yhat_scaled - y) * mask) / mask.sum() + acc.update(yhat_scaled[mask].view(-1).float(), y[mask].view(-1).float()) + epoch_loss_scaled += scaled_loss.item() + logger.info(f"example y: {y[0, :10].tolist()}") + logger.info(f"example yhat: {yhat_scaled[0, :10].tolist()}") + acc0 = acc.acc(tol=0) + acc1 = acc.acc(tol=1) + acc2 = acc.acc(tol=2) + acc3 = acc.acc(tol=3) + logger.info(f"accs: {acc0,acc1,acc2,acc3}") + return epoch_loss / len(loader), epoch_loss_scaled / len(loader), acc0, acc1, acc2, acc3 + + +@hydra.main(config_path=".", config_name="duration_predictor.yaml") +def main(cfg): + logger.info(f"{cfg}") + train(cfg) + + +if __name__ == "__main__": + main() diff --git a/data/fairseq/examples/emotion_conversion/emotion_models/duration_predictor.yaml b/data/fairseq/examples/emotion_conversion/emotion_models/duration_predictor.yaml new file mode 100644 index 0000000000000000000000000000000000000000..0e976f484351a7d64eaedb70eb7866ce8555b419 --- /dev/null +++ b/data/fairseq/examples/emotion_conversion/emotion_models/duration_predictor.yaml @@ -0,0 +1,48 @@ +train_tsv: "/denoising/emov/train.tsv" +train_km: "/denoising/emov/train.km" +valid_tsv: "/denoising/emov/valid.tsv" +valid_km: "/denoising/emov/valid.km" + +n_tokens: 200 +batch_size: 32 +lr: 0.0001 +epochs: 300 +model: "cnn" +substring: "" + +rnn: + _target_: emotion_models.duration_predictor.RnnPredictor + n_tokens: ${n_tokens} + emb_dim: 128 + rnn_hidden: 128 + output_dim: 1 + dropout: 0 + n_layers: 1 + +optimizer: + _target_: torch.optim.Adam + lr: ${lr} + betas: [0.9, 0.98] + eps: 0.000000001 + weight_decay: 0 + +cnn: + _target_: emotion_models.duration_predictor.CnnPredictor + n_tokens: ${n_tokens} + emb_dim: 128 + channels: 256 + kernel: 3 + output_dim: 1 + dropout: 0.5 + n_layers: 1 + +hydra: + run: + dir: /checkpoint/felixkreuk/experiments/duration_predictor/${hydra.job.override_dirname} + job: + config: + # configuration for the ${hydra.job.override_dirname} runtime variable + override_dirname: + kv_sep: '=' + item_sep: ',' + exclude_keys: ['train_tsv', 'train_km', 'valid_tsv', 'valid_km'] diff --git a/data/fairseq/examples/emotion_conversion/emotion_models/pitch_predictor.py b/data/fairseq/examples/emotion_conversion/emotion_models/pitch_predictor.py new file mode 100644 index 0000000000000000000000000000000000000000..431446996c30e1dadb61a1834e5ca56b647fe8f1 --- /dev/null +++ b/data/fairseq/examples/emotion_conversion/emotion_models/pitch_predictor.py @@ -0,0 +1,559 @@ +import logging +import os +import random +import sys +from collections import defaultdict + +import hydra +import numpy as np +import torch +import torch.nn as nn +import torch.nn.functional as F +from einops import rearrange +from einops.layers.torch import Rearrange +from scipy.io.wavfile import read +from scipy.ndimage import gaussian_filter1d +from torch.utils.data import DataLoader, Dataset +from tqdm import tqdm + +dir_path = os.path.dirname(__file__) +resynth_path = os.path.dirname(dir_path) + "/speech-resynthesis" +sys.path.append(resynth_path) +from dataset import parse_speaker, parse_style +from .utils import F0Stat + +MAX_WAV_VALUE = 32768.0 +logger = logging.getLogger(__name__) + + +def quantize_f0(speaker_to_f0, nbins, normalize, log): + f0_all = [] + for speaker, f0 in speaker_to_f0.items(): + f0 = f0.raw_data + if log: + f0 = f0.log() + mean = speaker_to_f0[speaker].mean_log if log else speaker_to_f0[speaker].mean + std = speaker_to_f0[speaker].std_log if log else speaker_to_f0[speaker].std + if normalize == "mean": + f0 = f0 - mean + elif normalize == "meanstd": + f0 = (f0 - mean) / std + f0_all.extend(f0.tolist()) + + hist, bin_x = np.histogram(f0_all, 100000) + cum_hist = np.cumsum(hist) / len(f0_all) * 100 + + bin_offset = [] + bin_size = 100 / nbins + threshold = bin_size + for i in range(nbins - 1): + index = (np.abs(cum_hist - threshold)).argmin() + bin_offset.append(bin_x[index]) + threshold += bin_size + bins = np.array(bin_offset) + bins = torch.FloatTensor(bins) + + return bins + + +def save_ckpt(model, path, model_class, f0_min, f0_max, f0_bins, speaker_stats): + ckpt = { + "state_dict": model.state_dict(), + "padding_token": model.padding_token, + "model_class": model_class, + "speaker_stats": speaker_stats, + "f0_min": f0_min, + "f0_max": f0_max, + "f0_bins": f0_bins, + } + torch.save(ckpt, path) + + +def load_ckpt(path): + ckpt = torch.load(path) + ckpt["model_class"]["_target_"] = "emotion_models.pitch_predictor.CnnPredictor" + model = hydra.utils.instantiate(ckpt["model_class"]) + model.load_state_dict(ckpt["state_dict"]) + model.setup_f0_stats( + ckpt["f0_min"], + ckpt["f0_max"], + ckpt["f0_bins"], + ckpt["speaker_stats"], + ) + return model + + +def freq2bin(f0, f0_min, f0_max, bins): + f0 = f0.clone() + f0[f0 < f0_min] = f0_min + f0[f0 > f0_max] = f0_max + f0 = torch.bucketize(f0, bins) + return f0 + + +def bin2freq(x, f0_min, f0_max, bins, mode): + n_bins = len(bins) + 1 + assert x.shape[-1] == n_bins + bins = torch.cat([torch.tensor([f0_min]), bins]).to(x.device) + if mode == "mean": + f0 = (x * bins).sum(-1, keepdims=True) / x.sum(-1, keepdims=True) + elif mode == "argmax": + idx = F.one_hot(x.argmax(-1), num_classes=n_bins) + f0 = (idx * bins).sum(-1, keepdims=True) + else: + raise NotImplementedError() + return f0[..., 0] + + +def load_wav(full_path): + sampling_rate, data = read(full_path) + return data, sampling_rate + + +def l1_loss(input, target): + return F.l1_loss(input=input.float(), target=target.float(), reduce=False) + + +def l2_loss(input, target): + return F.mse_loss(input=input.float(), target=target.float(), reduce=False) + + +class Collator: + def __init__(self, padding_idx): + self.padding_idx = padding_idx + + def __call__(self, batch): + tokens = [item[0] for item in batch] + lengths = [len(item) for item in tokens] + tokens = torch.nn.utils.rnn.pad_sequence( + tokens, batch_first=True, padding_value=self.padding_idx + ) + f0 = [item[1] for item in batch] + f0 = torch.nn.utils.rnn.pad_sequence( + f0, batch_first=True, padding_value=self.padding_idx + ) + f0_raw = [item[2] for item in batch] + f0_raw = torch.nn.utils.rnn.pad_sequence( + f0_raw, batch_first=True, padding_value=self.padding_idx + ) + spk = [item[3] for item in batch] + spk = torch.LongTensor(spk) + gst = [item[4] for item in batch] + gst = torch.LongTensor(gst) + mask = tokens != self.padding_idx + return tokens, f0, f0_raw, spk, gst, mask, lengths + + +class CnnPredictor(nn.Module): + def __init__( + self, + n_tokens, + emb_dim, + channels, + kernel, + dropout, + n_layers, + spk_emb, + gst_emb, + n_bins, + f0_pred, + f0_log, + f0_norm, + ): + super(CnnPredictor, self).__init__() + self.n_tokens = n_tokens + self.emb_dim = emb_dim + self.f0_log = f0_log + self.f0_pred = f0_pred + self.padding_token = n_tokens + self.f0_norm = f0_norm + # add 1 extra embedding for padding token, set the padding index to be the last token + # (tokens from the clustering start at index 0) + self.token_emb = nn.Embedding( + n_tokens + 1, emb_dim, padding_idx=self.padding_token + ) + + self.spk_emb = spk_emb + self.gst_emb = nn.Embedding(20, gst_emb) + self.setup = False + + feats = emb_dim + gst_emb + # feats = emb_dim + gst_emb + (256 if spk_emb else 0) + layers = [ + nn.Sequential( + Rearrange("b t c -> b c t"), + nn.Conv1d( + feats, channels, kernel_size=kernel, padding=(kernel - 1) // 2 + ), + Rearrange("b c t -> b t c"), + nn.ReLU(), + nn.LayerNorm(channels), + nn.Dropout(dropout), + ) + ] + for _ in range(n_layers - 1): + layers += [ + nn.Sequential( + Rearrange("b t c -> b c t"), + nn.Conv1d( + channels, + channels, + kernel_size=kernel, + padding=(kernel - 1) // 2, + ), + Rearrange("b c t -> b t c"), + nn.ReLU(), + nn.LayerNorm(channels), + nn.Dropout(dropout), + ) + ] + self.conv_layer = nn.ModuleList(layers) + self.proj = nn.Linear(channels, n_bins) + + def forward(self, x, gst=None): + x = self.token_emb(x) + feats = [x] + + if gst is not None: + gst = self.gst_emb(gst) + gst = rearrange(gst, "b c -> b c 1") + gst = F.interpolate(gst, x.shape[1]) + gst = rearrange(gst, "b c t -> b t c") + feats.append(gst) + + x = torch.cat(feats, dim=-1) + + for i, conv in enumerate(self.conv_layer): + if i != 0: + x = conv(x) + x + else: + x = conv(x) + + x = self.proj(x) + x = x.squeeze(-1) + + if self.f0_pred == "mean": + x = torch.sigmoid(x) + elif self.f0_pred == "argmax": + x = torch.softmax(x, dim=-1) + else: + raise NotImplementedError + return x + + def setup_f0_stats(self, f0_min, f0_max, f0_bins, speaker_stats): + self.f0_min = f0_min + self.f0_max = f0_max + self.f0_bins = f0_bins + self.speaker_stats = speaker_stats + self.setup = True + + def inference(self, x, spk_id=None, gst=None): + assert ( + self.setup == True + ), "make sure that `setup_f0_stats` was called before inference!" + probs = self(x, gst) + f0 = bin2freq(probs, self.f0_min, self.f0_max, self.f0_bins, self.f0_pred) + for i in range(f0.shape[0]): + mean = ( + self.speaker_stats[spk_id[i].item()].mean_log + if self.f0_log + else self.speaker_stats[spk_id[i].item()].mean + ) + std = ( + self.speaker_stats[spk_id[i].item()].std_log + if self.f0_log + else self.speaker_stats[spk_id[i].item()].std + ) + if self.f0_norm == "mean": + f0[i] = f0[i] + mean + if self.f0_norm == "meanstd": + f0[i] = (f0[i] * std) + mean + if self.f0_log: + f0 = f0.exp() + return f0 + + +class PitchDataset(Dataset): + def __init__( + self, + tsv_path, + km_path, + substring, + spk, + spk2id, + gst, + gst2id, + f0_bins, + f0_bin_type, + f0_smoothing, + f0_norm, + f0_log, + ): + lines = open(tsv_path, "r").readlines() + self.root, self.tsv = lines[0], lines[1:] + self.root = self.root.strip() + self.km = open(km_path, "r").readlines() + print(f"loaded {len(self.km)} files") + + self.spk = spk + self.spk2id = spk2id + self.gst = gst + self.gst2id = gst2id + + self.f0_bins = f0_bins + self.f0_smoothing = f0_smoothing + self.f0_norm = f0_norm + self.f0_log = f0_log + + if substring != "": + tsv, km = [], [] + for tsv_line, km_line in zip(self.tsv, self.km): + if substring.lower() in tsv_line.lower(): + tsv.append(tsv_line) + km.append(km_line) + self.tsv, self.km = tsv, km + print(f"after filtering: {len(self.km)} files") + + self.speaker_stats = self._compute_f0_stats() + self.f0_min, self.f0_max = self._compute_f0_minmax() + if f0_bin_type == "adaptive": + self.f0_bins = quantize_f0( + self.speaker_stats, self.f0_bins, self.f0_norm, self.f0_log + ) + elif f0_bin_type == "uniform": + self.f0_bins = torch.linspace(self.f0_min, self.f0_max, self.f0_bins + 1)[ + 1:-1 + ] + else: + raise NotImplementedError + print(f"f0 min: {self.f0_min}, f0 max: {self.f0_max}") + print(f"bins: {self.f0_bins} (shape: {self.f0_bins.shape})") + + def __len__(self): + return len(self.km) + + def _load_f0(self, tsv_line): + tsv_line = tsv_line.split("\t")[0] + f0 = self.root + "/" + tsv_line.replace(".wav", ".yaapt.f0.npy") + f0 = np.load(f0) + f0 = torch.FloatTensor(f0) + return f0 + + def _preprocess_f0(self, f0, spk): + mask = f0 != -999999 # process all frames + # mask = (f0 != 0) # only process voiced frames + mean = ( + self.speaker_stats[spk].mean_log + if self.f0_log + else self.speaker_stats[spk].mean + ) + std = ( + self.speaker_stats[spk].std_log + if self.f0_log + else self.speaker_stats[spk].std + ) + if self.f0_log: + f0[f0 == 0] = 1e-5 + f0[mask] = f0[mask].log() + if self.f0_norm == "mean": + f0[mask] = f0[mask] - mean + if self.f0_norm == "meanstd": + f0[mask] = (f0[mask] - mean) / std + return f0 + + def _compute_f0_minmax(self): + f0_min, f0_max = float("inf"), -float("inf") + for tsv_line in tqdm(self.tsv, desc="computing f0 minmax"): + spk = self.spk2id[parse_speaker(tsv_line, self.spk)] + f0 = self._load_f0(tsv_line) + f0 = self._preprocess_f0(f0, spk) + f0_min = min(f0_min, f0.min().item()) + f0_max = max(f0_max, f0.max().item()) + return f0_min, f0_max + + def _compute_f0_stats(self): + from functools import partial + + speaker_stats = defaultdict(partial(F0Stat, True)) + for tsv_line in tqdm(self.tsv, desc="computing speaker stats"): + spk = self.spk2id[parse_speaker(tsv_line, self.spk)] + f0 = self._load_f0(tsv_line) + mask = f0 != 0 + f0 = f0[mask] # compute stats only on voiced parts + speaker_stats[spk].update(f0) + return speaker_stats + + def __getitem__(self, i): + x = self.km[i] + x = x.split(" ") + x = list(map(int, x)) + x = torch.LongTensor(x) + + gst = parse_style(self.tsv[i], self.gst) + gst = self.gst2id[gst] + spk = parse_speaker(self.tsv[i], self.spk) + spk = self.spk2id[spk] + + f0_raw = self._load_f0(self.tsv[i]) + f0 = self._preprocess_f0(f0_raw.clone(), spk) + + f0 = F.interpolate(f0.unsqueeze(0).unsqueeze(0), x.shape[0])[0, 0] + f0_raw = F.interpolate(f0_raw.unsqueeze(0).unsqueeze(0), x.shape[0])[0, 0] + + f0 = freq2bin(f0, f0_min=self.f0_min, f0_max=self.f0_max, bins=self.f0_bins) + f0 = F.one_hot(f0.long(), num_classes=len(self.f0_bins) + 1).float() + if self.f0_smoothing > 0: + f0 = torch.tensor( + gaussian_filter1d(f0.float().numpy(), sigma=self.f0_smoothing) + ) + return x, f0, f0_raw, spk, gst + + +def train(cfg): + device = "cuda:0" + # add 1 extra embedding for padding token, set the padding index to be the last token + # (tokens from the clustering start at index 0) + padding_token = cfg.n_tokens + collate_fn = Collator(padding_idx=padding_token) + train_ds = PitchDataset( + cfg.train_tsv, + cfg.train_km, + substring=cfg.substring, + spk=cfg.spk, + spk2id=cfg.spk2id, + gst=cfg.gst, + gst2id=cfg.gst2id, + f0_bins=cfg.f0_bins, + f0_bin_type=cfg.f0_bin_type, + f0_smoothing=cfg.f0_smoothing, + f0_norm=cfg.f0_norm, + f0_log=cfg.f0_log, + ) + valid_ds = PitchDataset( + cfg.valid_tsv, + cfg.valid_km, + substring=cfg.substring, + spk=cfg.spk, + spk2id=cfg.spk2id, + gst=cfg.gst, + gst2id=cfg.gst2id, + f0_bins=cfg.f0_bins, + f0_bin_type=cfg.f0_bin_type, + f0_smoothing=cfg.f0_smoothing, + f0_norm=cfg.f0_norm, + f0_log=cfg.f0_log, + ) + train_dl = DataLoader( + train_ds, + num_workers=0, + batch_size=cfg.batch_size, + shuffle=True, + collate_fn=collate_fn, + ) + valid_dl = DataLoader( + valid_ds, num_workers=0, batch_size=16, shuffle=False, collate_fn=collate_fn + ) + + f0_min = train_ds.f0_min + f0_max = train_ds.f0_max + f0_bins = train_ds.f0_bins + speaker_stats = train_ds.speaker_stats + + model = hydra.utils.instantiate(cfg["model"]).to(device) + model.setup_f0_stats(f0_min, f0_max, f0_bins, speaker_stats) + + optimizer = hydra.utils.instantiate(cfg.optimizer, model.parameters()) + + best_loss = float("inf") + for epoch in range(cfg.epochs): + train_loss, train_l2_loss, train_l2_voiced_loss = run_epoch( + model, train_dl, optimizer, device, cfg, mode="train" + ) + valid_loss, valid_l2_loss, valid_l2_voiced_loss = run_epoch( + model, valid_dl, None, device, cfg, mode="valid" + ) + print( + f"[epoch {epoch}] train loss: {train_loss:.3f}, l2 loss: {train_l2_loss:.3f}, l2 voiced loss: {train_l2_voiced_loss:.3f}" + ) + print( + f"[epoch {epoch}] valid loss: {valid_loss:.3f}, l2 loss: {valid_l2_loss:.3f}, l2 voiced loss: {valid_l2_voiced_loss:.3f}" + ) + if valid_l2_voiced_loss < best_loss: + path = f"{os.getcwd()}/pitch_predictor.ckpt" + save_ckpt(model, path, cfg["model"], f0_min, f0_max, f0_bins, speaker_stats) + best_loss = valid_l2_voiced_loss + print(f"saved checkpoint: {path}") + print(f"[epoch {epoch}] best loss: {best_loss:.3f}") + + +def run_epoch(model, loader, optimizer, device, cfg, mode): + if mode == "train": + model.train() + else: + model.eval() + + epoch_loss = 0 + l1 = 0 + l1_voiced = 0 + for x, f0_bin, f0_raw, spk_id, gst, mask, _ in tqdm(loader): + x, f0_bin, f0_raw, spk_id, gst, mask = ( + x.to(device), + f0_bin.to(device), + f0_raw.to(device), + spk_id.to(device), + gst.to(device), + mask.to(device), + ) + b, t, n_bins = f0_bin.shape + yhat = model(x, gst) + nonzero_mask = (f0_raw != 0).logical_and(mask) + yhat_raw = model.inference(x, spk_id, gst) + expanded_mask = mask.unsqueeze(-1).expand(-1, -1, n_bins) + if cfg.f0_pred == "mean": + loss = F.binary_cross_entropy( + yhat[expanded_mask], f0_bin[expanded_mask] + ).mean() + elif cfg.f0_pred == "argmax": + loss = F.cross_entropy( + rearrange(yhat, "b t d -> (b t) d"), + rearrange(f0_bin.argmax(-1), "b t -> (b t)"), + reduce=False, + ) + loss = rearrange(loss, "(b t) -> b t", b=b, t=t) + loss = (loss * mask).sum() / mask.float().sum() + else: + raise NotImplementedError + l1 += F.l1_loss(yhat_raw[mask], f0_raw[mask]).item() + l1_voiced += F.l1_loss(yhat_raw[nonzero_mask], f0_raw[nonzero_mask]).item() + epoch_loss += loss.item() + + if mode == "train": + loss.backward() + nn.utils.clip_grad_norm_(model.parameters(), 1.0) + optimizer.step() + + print(f"{mode} example y: {f0_bin.argmax(-1)[0, 50:60].tolist()}") + print(f"{mode} example yhat: {yhat.argmax(-1)[0, 50:60].tolist()}") + print(f"{mode} example y: {f0_raw[0, 50:60].round().tolist()}") + print(f"{mode} example yhat: {yhat_raw[0, 50:60].round().tolist()}") + return epoch_loss / len(loader), l1 / len(loader), l1_voiced / len(loader) + + +@hydra.main(config_path=dir_path, config_name="pitch_predictor.yaml") +def main(cfg): + np.random.seed(1) + random.seed(1) + torch.manual_seed(1) + from hydra.core.hydra_config import HydraConfig + + overrides = { + x.split("=")[0]: x.split("=")[1] + for x in HydraConfig.get().overrides.task + if "/" not in x + } + print(f"{cfg}") + train(cfg) + + +if __name__ == "__main__": + main() diff --git a/data/fairseq/examples/emotion_conversion/emotion_models/pitch_predictor.yaml b/data/fairseq/examples/emotion_conversion/emotion_models/pitch_predictor.yaml new file mode 100644 index 0000000000000000000000000000000000000000..d2dbb862c358a0af58c0c4abf37fbdb537da6b44 --- /dev/null +++ b/data/fairseq/examples/emotion_conversion/emotion_models/pitch_predictor.yaml @@ -0,0 +1,64 @@ +train_tsv: "/denoising/emov/train.tsv" +train_km: "/denoising/emov/train.km" +valid_tsv: "/denoising/emov/valid.tsv" +valid_km: "/denoising/emov/valid.km" + +n_tokens: 200 +batch_size: 64 +lr: 0.0001 +epochs: 1000 + +substring: "" +loss: "l2" +spk: "parent_parent_name" +gst: "emotion" + +f0_bins: 50 +f0_pred: "mean" # [argmax, mean] +f0_smoothing: 0.1 +f0_norm: "mean" +f0_log: false +f0_bin_type: "adaptive" # [uniform, adaptive] + +spk2id: + bea: 0 + jenie: 1 + josh: 2 + sam: 3 + +gst2id: + amused: 0 + angry: 1 + disgusted: 2 + neutral: 3 + sleepy: 4 + +optimizer: + _target_: torch.optim.Adam + lr: ${lr} + +model: + _target_: emotion_models.pitch_predictor.CnnPredictor + n_tokens: ${n_tokens} + emb_dim: 256 + channels: 256 + kernel: 5 + dropout: 0.1 + n_layers: 6 + spk_emb: true + gst_emb: 8 + n_bins: ${f0_bins} + f0_pred: ${f0_pred} + f0_log: ${f0_log} + f0_norm: ${f0_norm} + +hydra: + run: + dir: /checkpoint/felixkreuk/experiments/pitch_predictor/${hydra.job.override_dirname} + job: + config: + # configuration for the ${hydra.job.override_dirname} runtime variable + override_dirname: + kv_sep: '=' + item_sep: ',' + exclude_keys: ['train_tsv', 'train_km', 'valid_tsv', 'valid_km'] diff --git a/data/fairseq/examples/emotion_conversion/emotion_models/utils.py b/data/fairseq/examples/emotion_conversion/emotion_models/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..4199c310f8ee6e50c81ca0645ea30cc672fd0b5c --- /dev/null +++ b/data/fairseq/examples/emotion_conversion/emotion_models/utils.py @@ -0,0 +1,78 @@ +import torch + + +class Stat: + def __init__(self, keep_raw=False): + self.x = 0.0 + self.x2 = 0.0 + self.z = 0.0 # z = logx + self.z2 = 0.0 + self.n = 0.0 + self.u = 0.0 + self.keep_raw = keep_raw + self.raw = [] + + def update(self, new_x): + new_z = new_x.log() + + self.x += new_x.sum() + self.x2 += (new_x**2).sum() + self.z += new_z.sum() + self.z2 += (new_z**2).sum() + self.n += len(new_x) + self.u += 1 + + if self.keep_raw: + self.raw.append(new_x) + + @property + def mean(self): + return self.x / self.n + + @property + def std(self): + return (self.x2 / self.n - self.mean**2) ** 0.5 + + @property + def mean_log(self): + return self.z / self.n + + @property + def std_log(self): + return (self.z2 / self.n - self.mean_log**2) ** 0.5 + + @property + def n_frms(self): + return self.n + + @property + def n_utts(self): + return self.u + + @property + def raw_data(self): + assert self.keep_raw, "does not support storing raw data!" + return torch.cat(self.raw) + + +class F0Stat(Stat): + def update(self, new_x): + # assume unvoiced frames are 0 and consider only voiced frames + if new_x is not None: + super().update(new_x[new_x != 0]) + + +class Accuracy: + def __init__(self): + self.y, self.yhat = [], [] + + def update(self, yhat, y): + self.yhat.append(yhat) + self.y.append(y) + + def acc(self, tol): + yhat = torch.cat(self.yhat) + y = torch.cat(self.y) + acc = torch.abs(yhat - y) <= tol + acc = acc.float().mean().item() + return acc diff --git a/data/fairseq/examples/emotion_conversion/fairseq_models/__init__.py b/data/fairseq/examples/emotion_conversion/fairseq_models/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..441bc03db49c78556796db5d0b680b0cc7a99ed7 --- /dev/null +++ b/data/fairseq/examples/emotion_conversion/fairseq_models/__init__.py @@ -0,0 +1,226 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +from fairseq import utils +from fairseq.models import ( + FairseqMultiModel, + register_model, + register_model_architecture, +) +from fairseq.models.transformer import ( + Embedding, + base_architecture, +) +from fairseq.models.multilingual_transformer import ( + MultilingualTransformerModel, + base_multilingual_architecture, +) +from fairseq.utils import safe_hasattr +from collections import OrderedDict + + +@register_model("multilingual_transformer_from_mbart") +class MultilingualTransformerModelFromMbart(MultilingualTransformerModel): + @classmethod + def build_model(cls, args, task): + """Build a new model instance.""" + from fairseq.tasks.multilingual_translation import MultilingualTranslationTask + + assert isinstance(task, MultilingualTranslationTask) + + # make sure all arguments are present in older models + base_multilingual_architecture(args) + + if not safe_hasattr(args, "max_source_positions"): + args.max_source_positions = 1024 + if not safe_hasattr(args, "max_target_positions"): + args.max_target_positions = 1024 + + src_langs = [lang_pair.split("-")[0] for lang_pair in task.model_lang_pairs] + tgt_langs = [lang_pair.split("-")[1] for lang_pair in task.model_lang_pairs] + + if args.share_encoders: + args.share_encoder_embeddings = True + if args.share_decoders: + args.share_decoder_embeddings = True + + def build_embedding(dictionary, embed_dim, path=None): + num_embeddings = len(dictionary) + padding_idx = dictionary.pad() + emb = Embedding(num_embeddings, embed_dim, padding_idx) + # if provided, load from preloaded dictionaries + if path: + embed_dict = utils.parse_embedding(path) + utils.load_embedding(embed_dict, dictionary, emb) + return emb + + # build shared embeddings (if applicable) + shared_encoder_embed_tokens, shared_decoder_embed_tokens = None, None + if args.share_all_embeddings: + if args.encoder_embed_dim != args.decoder_embed_dim: + raise ValueError( + "--share-all-embeddings requires --encoder-embed-dim to match --decoder-embed-dim" + ) + if args.decoder_embed_path and ( + args.decoder_embed_path != args.encoder_embed_path + ): + raise ValueError( + "--share-all-embeddings not compatible with --decoder-embed-path" + ) + shared_encoder_embed_tokens = FairseqMultiModel.build_shared_embeddings( + dicts=task.dicts, + langs=task.langs, + embed_dim=args.encoder_embed_dim, + build_embedding=build_embedding, + pretrained_embed_path=args.encoder_embed_path, + ) + shared_decoder_embed_tokens = shared_encoder_embed_tokens + args.share_decoder_input_output_embed = True + else: + if args.share_encoder_embeddings: + shared_encoder_embed_tokens = FairseqMultiModel.build_shared_embeddings( + dicts=task.dicts, + langs=src_langs, + embed_dim=args.encoder_embed_dim, + build_embedding=build_embedding, + pretrained_embed_path=args.encoder_embed_path, + ) + if args.share_decoder_embeddings: + shared_decoder_embed_tokens = FairseqMultiModel.build_shared_embeddings( + dicts=task.dicts, + langs=tgt_langs, + embed_dim=args.decoder_embed_dim, + build_embedding=build_embedding, + pretrained_embed_path=args.decoder_embed_path, + ) + + # encoders/decoders for each language + lang_encoders, lang_decoders = {}, {} + + def get_encoder(lang): + if lang not in lang_encoders: + if shared_encoder_embed_tokens is not None: + encoder_embed_tokens = shared_encoder_embed_tokens + else: + encoder_embed_tokens = build_embedding( + task.dicts[lang], + args.encoder_embed_dim, + args.encoder_embed_path, + ) + lang_encoders[lang] = MultilingualTransformerModel._get_module_class( + True, args, task.dicts[lang], encoder_embed_tokens, src_langs + ) + return lang_encoders[lang] + + def get_decoder(lang): + if lang not in lang_decoders: + if shared_decoder_embed_tokens is not None: + decoder_embed_tokens = shared_decoder_embed_tokens + else: + decoder_embed_tokens = build_embedding( + task.dicts[lang], + args.decoder_embed_dim, + args.decoder_embed_path, + ) + lang_decoders[lang] = MultilingualTransformerModel._get_module_class( + False, args, task.dicts[lang], decoder_embed_tokens, tgt_langs + ) + return lang_decoders[lang] + + # shared encoders/decoders (if applicable) + shared_encoder, shared_decoder = None, None + if args.share_encoders: + shared_encoder = get_encoder(src_langs[0]) + if args.share_decoders: + shared_decoder = get_decoder(tgt_langs[0]) + + encoders, decoders = OrderedDict(), OrderedDict() + for lang_pair, src, tgt in zip(task.model_lang_pairs, src_langs, tgt_langs): + encoders[lang_pair] = ( + shared_encoder if shared_encoder is not None else get_encoder(src) + ) + decoders[lang_pair] = ( + shared_decoder if shared_decoder is not None else get_decoder(tgt) + ) + + return MultilingualTransformerModelFromMbart(encoders, decoders) + + def load_state_dict(self, state_dict, strict=True, model_cfg=None): + state_dict_subset = state_dict.copy() + lang_pairs = set([x.split(".")[1] for x in state_dict.keys()]) + finetune_mode = not any("neutral" in lp for lp in lang_pairs) + + if finetune_mode: + # load a pre-trained mBART/BART model + # we need this code because mBART/BART are not of type FairseqMultiModel but FairseqModel + # so we hackishly load the weights by replicating them for all lang pairs + print("loading pre-trained BART") + self_state_dict = self.state_dict() + for k, v in state_dict.items(): + for lang_pair in self.models: + new_key = k if "models." in k else f"models.{lang_pair}.{k}" + # print(new_key) + if self_state_dict[new_key].shape == v.shape: + state_dict_subset[new_key] = v + elif any( + w in k + for w in [ + "encoder.embed_tokens.weight", + "decoder.embed_tokens.weight", + "decoder.output_projection.weight", + ] + ): + # why vocab_size - 5? because there are `vocab_size` tokens from the language + # and 5 additional tokens in the denoising task: eos,bos,pad,unk,mask. + # but in the translation task there are only `vocab_size` + 4 (no mask). + print( + f"{k}: {self_state_dict[new_key].shape} != {v.shape}", + end="", + flush=True, + ) + vocab_size = v.shape[0] - 5 + state_dict_subset[new_key] = self_state_dict[new_key] + state_dict_subset[new_key] = v[: vocab_size + 4] + print(f" => fixed by using first {vocab_size + 4} dims") + else: + raise ValueError("unable to load model due to mimatched dims!") + del state_dict_subset[k] + else: + print("loading pre-trained emotion translation model") + for k, _ in state_dict.items(): + assert k.startswith("models.") + lang_pair = k.split(".")[1] + if lang_pair not in self.models: + del state_dict_subset[k] + + super().load_state_dict(state_dict_subset, strict=strict, model_cfg=model_cfg) + + +@register_model_architecture("transformer", "transformer_small") +def transformer_small(args): + args.encoder_embed_dim = getattr(args, "encoder_embed_dim", 512) + args.encoder_ffn_embed_dim = getattr(args, "encoder_ffn_embed_dim", 512) + args.encoder_attention_heads = getattr(args, "encoder_attention_heads", 4) + args.encoder_layers = getattr(args, "encoder_layers", 3) + args.decoder_embed_dim = getattr(args, "decoder_embed_dim", 512) + args.decoder_ffn_embed_dim = getattr(args, "decoder_ffn_embed_dim", 512) + args.decoder_attention_heads = getattr(args, "decoder_attention_heads", 4) + args.decoder_layers = getattr(args, "decoder_layers", 3) + base_architecture(args) + + +@register_model_architecture( + "multilingual_transformer_from_mbart", "multilingual_small" +) +def multilingual_small(args): + args.encoder_embed_dim = getattr(args, "encoder_embed_dim", 512) + args.encoder_ffn_embed_dim = getattr(args, "encoder_ffn_embed_dim", 512) + args.encoder_attention_heads = getattr(args, "encoder_attention_heads", 4) + args.encoder_layers = getattr(args, "encoder_layers", 3) + args.decoder_embed_dim = getattr(args, "decoder_embed_dim", 512) + args.decoder_ffn_embed_dim = getattr(args, "decoder_ffn_embed_dim", 512) + args.decoder_attention_heads = getattr(args, "decoder_attention_heads", 4) + args.decoder_layers = getattr(args, "decoder_layers", 3) + base_multilingual_architecture(args) diff --git a/data/fairseq/examples/emotion_conversion/preprocess/__init__.py b/data/fairseq/examples/emotion_conversion/preprocess/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/data/fairseq/examples/emotion_conversion/preprocess/build_hifigan_manifest.py b/data/fairseq/examples/emotion_conversion/preprocess/build_hifigan_manifest.py new file mode 100644 index 0000000000000000000000000000000000000000..29c0d79ceeca7f03087781394b907b089197b4e9 --- /dev/null +++ b/data/fairseq/examples/emotion_conversion/preprocess/build_hifigan_manifest.py @@ -0,0 +1,38 @@ +import torchaudio +import argparse +import json + +def main(): + parser = argparse.ArgumentParser(description="example: python create_hifigan_manifest.py --tsv /checkpoint/felixkreuk/datasets/vctk/splits/vctk_16khz/train.tsv --km /checkpoint/felixkreuk/experiments/hubert/hubert_feats/vctk_16khz_km_100/train.km --km_type hubert_100km > ~/tmp/tmp_mani.txt") + parser.add_argument("--tsv", required=True, help="path to fairseq tsv file") + parser.add_argument("--km", required=True, help="path to a km file generated by HuBERT clustering") + parser.add_argument("--km_type", required=True, help="name of the codes in the output json (for example: 'cpc_100km')") + args = parser.parse_args() + + km_lines = open(args.km, "r").readlines() + tsv_lines = open(args.tsv, "r").readlines() + assert len(km_lines) == len(tsv_lines) - 1, "tsv and km files are not of the same length!" + + wav_root = tsv_lines[0].strip() + tsv_lines = tsv_lines[1:] + + for tsv_line, km_line in zip(tsv_lines, km_lines): + tsv_line, km_line = tsv_line.strip(), km_line.strip() + wav_basename, wav_num_frames = tsv_line.split("\t") + wav_path = wav_root + "/" + wav_basename + wav_info = torchaudio.info(wav_path) + assert int(wav_num_frames) == wav_info.num_frames, "tsv duration and actual duration don't match!" + wav_duration = wav_info.num_frames / wav_info.sample_rate + manifest_line = {"audio": wav_path, "duration": wav_duration, args.km_type: km_line} + print(json.dumps(manifest_line)) + +if __name__ == "__main__": + """ + usage: + python create_hifigan_manifest.py \ + --tsv /checkpoint/felixkreuk/datasets/vctk/manifests/vctk_16khz/valid.tsv \ + --km /checkpoint/felixkreuk/datasets/vctk/manifests/vctk_16khz/hubert_km_100/valid.km \ + --km_type hubert \ + > /checkpoint/felixkreuk/datasets/vctk/manifests/vctk_16khz/hubert_km_100/hifigan_valid_manifest.txt + """ + main() diff --git a/data/fairseq/examples/emotion_conversion/preprocess/build_translation_manifests.py b/data/fairseq/examples/emotion_conversion/preprocess/build_translation_manifests.py new file mode 100644 index 0000000000000000000000000000000000000000..d38454a71368146c44dc73f6f37037dca0dd026d --- /dev/null +++ b/data/fairseq/examples/emotion_conversion/preprocess/build_translation_manifests.py @@ -0,0 +1,258 @@ +from glob import glob +import argparse +from collections import defaultdict, Counter +from itertools import combinations, product, groupby +from pathlib import Path +import os +from sklearn.utils import shuffle +import numpy as np +import random +from shutil import copy +from subprocess import check_call + +np.random.seed(42) +random.seed(42) + + +def get_fname(s): + return s.split("\t")[0] + +def get_emotion(s): + return get_fname(s).split("_")[0].split("/")[1].lower() + +def get_utt_id(s): + return get_fname(s).split(".")[0].split("_")[-1] + +def dedup(seq): + """ >> remove_repetitions("1 2 2 3 100 2 2 1") + '1 2 3 100 2 1' """ + seq = seq.strip().split(" ") + result = seq[:1] + reps = [] + rep_counter = 1 + for k in seq[1:]: + if k != result[-1]: + result += [k] + reps += [rep_counter] + rep_counter = 1 + else: + rep_counter += 1 + reps += [rep_counter] + assert len(reps) == len(result) and sum(reps) == len(seq) + return " ".join(result) + "\n" #, reps + +def remove_under_k(seq, k): + """ remove tokens that repeat less then k times in a row + >> remove_under_k("a a a a b c c c", 1) ==> a a a a c c c """ + seq = seq.strip().split(" ") + result = [] + + freqs = [(k,len(list(g))) for k, g in groupby(seq)] + for c, f in freqs: + if f > k: + result += [c for _ in range(f)] + return " ".join(result) + "\n" #, reps + + +def call(cmd): + print(cmd) + check_call(cmd, shell=True) + + +def denoising_preprocess(path, lang, dict): + bin = 'fairseq-preprocess' + cmd = [ + bin, + f'--trainpref {path}/train.{lang} --validpref {path}/valid.{lang} --testpref {path}/test.{lang}', + f'--destdir {path}/tokenized/{lang}', + '--only-source', + '--task multilingual_denoising', + '--workers 40', + ] + if dict != "": + cmd += [f'--srcdict {dict}'] + cmd = " ".join(cmd) + call(cmd) + + +def translation_preprocess(path, src_lang, trg_lang, dict, only_train=False): + bin = 'fairseq-preprocess' + cmd = [ + bin, + f'--source-lang {src_lang} --target-lang {trg_lang}', + f'--trainpref {path}/train', + f'--destdir {path}/tokenized', + '--workers 40', + ] + if not only_train: + cmd += [f'--validpref {path}/valid --testpref {path}/test'] + if dict != "": + cmd += [ + f'--srcdict {dict}', + f'--tgtdict {dict}', + ] + cmd = " ".join(cmd) + call(cmd) + + +def load_tsv_km(tsv_path, km_path): + assert tsv_path.exists() and km_path.exists() + tsv_lines = open(tsv_path, "r").readlines() + root, tsv_lines = tsv_lines[0], tsv_lines[1:] + km_lines = open(km_path, "r").readlines() + assert len(tsv_lines) == len(km_lines), ".tsv and .km should be the same length!" + return root, tsv_lines, km_lines + + +def main(): + desc = """ + this script takes as input .tsv and .km files for EMOV dataset, and a pairs of emotions. + it generates parallel .tsv and .km files for these emotions. for exmaple: + ❯ python build_emov_translation_manifests.py \ + /checkpoint/felixkreuk/datasets/emov/manifests/emov_16khz/train.tsv \ + /checkpoint/felixkreuk/datasets/emov/manifests/emov_16khz/emov_16khz_km_100/train.km \ + ~/tmp/emov_pairs \ + --src-emotion amused --trg-emotion neutral \ + --dedup --shuffle --cross-speaker --dry-run + """ + parser = argparse.ArgumentParser(description=desc) + parser.add_argument("data", type=Path, help="path to a dir containing .tsv and .km files containing emov dataset") + parser.add_argument("output_path", type=Path, help="output directory with the manifests will be created") + parser.add_argument("-cs", "--cross-speaker", action='store_true', help="if set then translation will occur also between speakers, meaning the same sentence can be translated between different speakers (default: false)") + parser.add_argument("-dd", "--dedup", action='store_true', help="remove repeated tokens (example: 'aaabc=>abc')") + parser.add_argument("-sh", "--shuffle", action='store_true', help="shuffle the data") + parser.add_argument("-ae", "--autoencode", action='store_true', help="include training pairs from the same emotion (this includes examples of the same sentence uttered by different people and examples where the src and trg are the exact same seq)") + parser.add_argument("-dr", "--dry-run", action='store_true', help="don't write anything to disk") + parser.add_argument("-zs", "--zero-shot", action='store_true', help="if true, the denoising task will train on the same splits as the translation task (split by utterance id). if false, the denoising task will train on randomly sampled splits (not split by utterance id)") + parser.add_argument("--km-ext", default="km", help="") + parser.add_argument("--dict", default="/checkpoint/felixkreuk/datasets/emov/manifests/emov_16khz/fairseq.dict.txt", help="") + args = parser.parse_args() + SPEAKERS = ["bea", "jenie", "josh", "sam", "SAME"] + EMOTIONS = ['neutral', 'amused', 'angry', 'disgusted', 'sleepy'] + + suffix = "" + if args.cross_speaker: suffix += "_cross-speaker" + if args.dedup: suffix += "_dedup" + translation_suffix = "" + if args.autoencode: translation_suffix += "_autoencode" + denoising_suffix = "" + denoising_suffix += "_zeroshot" if args.zero_shot else "_nonzeroshot" + + translation_dir = Path(args.output_path) / ("emov_multilingual_translation" + suffix + translation_suffix) + os.makedirs(translation_dir, exist_ok=True) + denoising_dir = Path(args.output_path) / ("emov_multilingual_denoising" + suffix + denoising_suffix) + os.makedirs(denoising_dir, exist_ok=True) + + denoising_data = [p.name for p in (args.data / "denoising").glob("*") if "emov" not in p.name] + + for split in ["train", "valid", "test"]: + root, tsv_lines, km_lines = load_tsv_km( + tsv_path = args.data / "denoising" / "emov" / f"{split}.tsv", + km_path = args.data / "denoising" / "emov" / f"{split}.{args.km_ext}" + ) + + # generate data for the multilingual denoising task + for EMOTION in EMOTIONS: + print("---") + print(split) + print(f"denoising: {EMOTION}") + emotion_tsv, emotion_km = [], [] + for tsv_line, km_line in zip(tsv_lines, km_lines): + if EMOTION.lower() in tsv_line.lower(): + km_line = km_line if not args.dedup else dedup(km_line) + emotion_tsv.append(tsv_line) + emotion_km.append(km_line) + print(f"{len(emotion_km)} samples") + open(denoising_dir / f"files.{split}.{EMOTION}", "w").writelines([root] + emotion_tsv) + open(denoising_dir / f"{split}.{EMOTION}", "w").writelines(emotion_km) + + for data in denoising_data: + with open(args.data / "denoising" / data / f"{split}.{args.km_ext}", "r") as f1: + with open(denoising_dir / f"{split}.{data}", "w") as f2: + f2.writelines([l if not args.dedup else dedup(l) for l in f1.readlines()]) + + # start of translation preprocessing + root, tsv_lines, km_lines = load_tsv_km( + tsv_path = args.data / "translation" / f"{split}.tsv", + km_path = args.data / "translation" / f"{split}.{args.km_ext}" + ) + + # generate data for the multilingual translation task + for SRC_EMOTION in EMOTIONS: + TRG_EMOTIONS = EMOTIONS if args.autoencode else set(EMOTIONS) - set([SRC_EMOTION]) + for TRG_EMOTION in TRG_EMOTIONS: + # when translating back to the same emotion - we dont want these emotion + # pairs to be part of the validation/test sets (because its not really emotion conversino) + # if SRC_EMOTION == TRG_EMOTION and split in ["valid", "test"]: continue + print("---") + print(split) + print(f"src emotions: {SRC_EMOTION}\ntrg emotions: {TRG_EMOTION}") + + # create a dictionary with the following structure: + # output[SPEAKER][UTT_ID] = list with indexes of line from the tsv file + # that match the speaker and utterance id. for exmaple: + # output = {'sam': {'0493': [875, 1608, 1822], ...}, ...} + # meaning, for speaker 'sam', utterance id '0493', the indexes in tsv_lines + # are 875, 1608, 1822 + spkr2utts = defaultdict(lambda: defaultdict(list)) + for i, tsv_line in enumerate(tsv_lines): + speaker = tsv_line.split("/")[0] + if args.cross_speaker: speaker = "SAME" + assert speaker in SPEAKERS, "unknown speaker! make sure the .tsv contains EMOV data" + utt_id = get_utt_id(tsv_line) + spkr2utts[speaker][utt_id].append(i) + + # create a tsv and km files with all the combinations for translation + src_tsv, trg_tsv, src_km, trg_km = [], [], [], [] + for speaker, utt_ids in spkr2utts.items(): + for utt_id, indices in utt_ids.items(): + # generate all pairs + pairs = [(x,y) for x in indices for y in indices] + # self-translation + if SRC_EMOTION == TRG_EMOTION: + pairs = [(x,y) for (x,y) in pairs if x == y] + # filter according to src and trg emotions + pairs = [(x,y) for (x,y) in pairs + if get_emotion(tsv_lines[x]) == SRC_EMOTION and get_emotion(tsv_lines[y]) == TRG_EMOTION] + + for idx1, idx2 in pairs: + assert get_utt_id(tsv_lines[idx1]) == get_utt_id(tsv_lines[idx2]) + src_tsv.append(tsv_lines[idx1]) + trg_tsv.append(tsv_lines[idx2]) + km_line_idx1 = km_lines[idx1] + km_line_idx2 = km_lines[idx2] + km_line_idx1 = km_line_idx1 if not args.dedup else dedup(km_line_idx1) + km_line_idx2 = km_line_idx2 if not args.dedup else dedup(km_line_idx2) + src_km.append(km_line_idx1) + trg_km.append(km_line_idx2) + assert len(src_tsv) == len(trg_tsv) == len(src_km) == len(trg_km) + print(f"{len(src_tsv)} pairs") + + if len(src_tsv) == 0: + raise Exception("ERROR: generated 0 pairs!") + + if args.dry_run: continue + + # create files + os.makedirs(translation_dir / f"{SRC_EMOTION}-{TRG_EMOTION}", exist_ok=True) + open(translation_dir / f"{SRC_EMOTION}-{TRG_EMOTION}" / f"files.{split}.{SRC_EMOTION}", "w").writelines([root] + src_tsv) + open(translation_dir / f"{SRC_EMOTION}-{TRG_EMOTION}" / f"files.{split}.{TRG_EMOTION}", "w").writelines([root] + trg_tsv) + open(translation_dir / f"{SRC_EMOTION}-{TRG_EMOTION}" / f"{split}.{SRC_EMOTION}", "w").writelines(src_km) + open(translation_dir / f"{SRC_EMOTION}-{TRG_EMOTION}" / f"{split}.{TRG_EMOTION}", "w").writelines(trg_km) + + + # fairseq-preprocess the denoising data + for EMOTION in EMOTIONS + denoising_data: + denoising_preprocess(denoising_dir, EMOTION, args.dict) + os.system(f"cp {args.dict} {denoising_dir}/tokenized/dict.txt") + + # fairseq-preprocess the translation data + os.makedirs(translation_dir / "tokenized", exist_ok=True) + for SRC_EMOTION in EMOTIONS: + TRG_EMOTIONS = EMOTIONS if args.autoencode else set(EMOTIONS) - set([SRC_EMOTION]) + for TRG_EMOTION in TRG_EMOTIONS: + translation_preprocess(translation_dir / f"{SRC_EMOTION}-{TRG_EMOTION}", SRC_EMOTION, TRG_EMOTION, args.dict)#, only_train=SRC_EMOTION==TRG_EMOTION) + os.system(f"cp -rf {translation_dir}/**/tokenized/* {translation_dir}/tokenized") + +if __name__ == "__main__": + main() diff --git a/data/fairseq/examples/emotion_conversion/preprocess/create_core_manifest.py b/data/fairseq/examples/emotion_conversion/preprocess/create_core_manifest.py new file mode 100644 index 0000000000000000000000000000000000000000..b55740e00becb6fc45d2d3c5396dc6ca85790e2a --- /dev/null +++ b/data/fairseq/examples/emotion_conversion/preprocess/create_core_manifest.py @@ -0,0 +1,91 @@ +from pathlib import Path +import os +import sys +import subprocess +import argparse +from datetime import datetime +import logging + +logging.basicConfig( + level=logging.INFO, + format='%(asctime)s [%(levelname)s] %(message)s', + handlers=[logging.FileHandler('debug.log'), logging.StreamHandler()] +) +logger = logging.getLogger(__name__) + + +def verify_dict_size(km, dict): + logger.info(f"verifying: {km}") + dict_size = len(open(dict, "r").readlines()) + km_vocab = set(open(km, "r").read().replace("\n", " ").split(" ")) + if "" in km_vocab: km_vocab.remove("") + km_vocab_size = len(km_vocab) + return dict_size == km_vocab_size + + +def verify_files_exist(l): + for f in l: + if not f.exists(): + logging.error(f"{f} doesn't exist!") + return False + return True + + +def run_cmd(cmd, print_output=True): + try: + out = subprocess.check_output(cmd, stderr=subprocess.STDOUT, universal_newlines=True, shell=True) + if print_output: + logger.info(f"command output:\n{out}") + return out + except subprocess.CalledProcessError as grepexc: + logger.info(f"error executing command!:\n{cmd}") + logger.info(grepexc.output) + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--tsv", default="/checkpoint/felixkreuk/datasets/emov/manifests/emov_16khz/data.tsv", type=Path) + parser.add_argument("--emov-km", required=True, type=Path) + parser.add_argument("--km", nargs='+', required=True, type=Path) + parser.add_argument("--seed", type=int, default=1) + parser.add_argument("--dict", default="/checkpoint/felixkreuk/datasets/emov/manifests/emov_16khz/fairseq.dict.txt") + parser.add_argument("--manifests-dir", type=Path, default="/checkpoint/felixkreuk/datasets/emov/manifests/emov_16khz") + args = parser.parse_args() + + manifests_dir = args.manifests_dir + date = datetime.now().strftime('%d%m%y') + outdir = manifests_dir / f"{date}" + + # verify input and create folders + all_kms = args.km + [args.emov_km] + assert verify_files_exist(all_kms), "make sure the km dir contains: train-clean-all.km, blizzard2013.km, data.km" + for codes in all_kms: + assert verify_dict_size(codes, args.dict), "dict argument doesn't match the vocabulary of the km file!" + assert not outdir.exists(), "data dir already exists!" + outdir.mkdir(parents=True, exist_ok=True) + + logger.info("generating denoising split (emov)") + run_cmd(f"python preprocess/split_km_tsv.py {args.tsv} {args.emov_km} --destdir {outdir}/denoising/emov -sh --seed {args.seed}") + for codes in args.km: + codes_name = os.path.basename(codes) + run_cmd(f"python preprocess/split_km.py {codes} --destdir {outdir}/denoising/{codes_name} -sh --seed {args.seed}") + + logger.info("generating translation split") + run_cmd(f"python preprocess/split_emov_km_tsv_by_uttid.py {args.tsv} {args.emov_km} --destdir {outdir}/translation --seed {args.seed}") + + emov_code_name = os.path.basename(args.emov_km) + logger.info("generating hifigan split") + run_cmd( + f"mkdir -p {outdir}/hifigan &&" + f"python preprocess/build_hifigan_manifest.py --km_type hubert --tsv {outdir}/denoising/emov/train.tsv --km {outdir}/denoising/emov/train.km > {outdir}/hifigan/train.txt &&" + f"python preprocess/build_hifigan_manifest.py --km_type hubert --tsv {outdir}/denoising/emov/valid.tsv --km {outdir}/denoising/emov/valid.km > {outdir}/hifigan/valid.txt &&" + f"python preprocess/build_hifigan_manifest.py --km_type hubert --tsv {outdir}/denoising/emov/test.tsv --km {outdir}/denoising/emov/test.km > {outdir}/hifigan/test.txt" + ) + + logger.info("generating fairseq manifests") + run_cmd(f"python preprocess/build_translation_manifests.py {outdir} {outdir}/fairseq-data -dd -cs --dict {args.dict}") + + logger.info(f"finished processing data at:\n{outdir}") + + +if __name__ == "__main__": + main() diff --git a/data/fairseq/examples/emotion_conversion/preprocess/extract_f0.py b/data/fairseq/examples/emotion_conversion/preprocess/extract_f0.py new file mode 100644 index 0000000000000000000000000000000000000000..4204aa4db1c39fac6a12a865741b5acca702230f --- /dev/null +++ b/data/fairseq/examples/emotion_conversion/preprocess/extract_f0.py @@ -0,0 +1,57 @@ +import argparse +from tqdm import tqdm +from multiprocessing import Manager, Pool + +from scipy.io.wavfile import read +from librosa.util import normalize +import numpy as np +import amfm_decompy.pYAAPT as pYAAPT +import amfm_decompy.basic_tools as basic + +MAX_WAV_VALUE = 32768.0 + +parser = argparse.ArgumentParser(description="") +parser.add_argument("tsv", help="") +parser.add_argument("--extractor", choices=["crepe", "pyaapt"], default="pyaapt", help="") +parser.add_argument("--interp", action="store_true", help="") +parser.add_argument("--n_workers", type=int, default=40, help="") +args = parser.parse_args() + +tsv_lines = open(args.tsv, "r").readlines() +root, tsv_lines = tsv_lines[0].strip(), tsv_lines[1:] + + +def extract_f0(tsv_line): + wav_path, _ = tsv_line.split("\t") + wav_path = root.strip() + "/" + wav_path + sr, wav = read(wav_path) + wav = wav / MAX_WAV_VALUE + wav = normalize(wav) * 0.95 + + if args.extractor == "pyaapt": + frame_length = 20.0 + pad = int(frame_length / 1000 * sr) // 2 + wav = np.pad(wav.squeeze(), (pad, pad), "constant", constant_values=0) + signal = basic.SignalObj(wav, sr) + pitch = pYAAPT.yaapt( + signal, + **{ + 'frame_length': frame_length, + 'frame_space': 5.0, + 'nccf_thresh1': 0.25, + 'tda_frame_length': 25.0 + }) + pitch = pitch.samp_interp[None, None, :] if args.interp else pitch.samp_values[None, None, :] + pitch = pitch[0, 0] + f0_path = wav_path.replace(".wav", ".yaapt") + f0_path += ".interp.f0" if args.interp else ".f0" + np.save(f0_path, pitch) + + +def main(): + with Pool(args.n_workers) as p: + r = list(tqdm(p.imap(extract_f0, tsv_lines), total=len(tsv_lines))) + + +if __name__ == "__main__": + main() diff --git a/data/fairseq/examples/emotion_conversion/preprocess/process_km.py b/data/fairseq/examples/emotion_conversion/preprocess/process_km.py new file mode 100644 index 0000000000000000000000000000000000000000..864a022105f3c6f82b53a4553a438e76e211f40d --- /dev/null +++ b/data/fairseq/examples/emotion_conversion/preprocess/process_km.py @@ -0,0 +1,40 @@ +import sys +import argparse +from tqdm import tqdm +from build_emov_translation_manifests import dedup, remove_under_k + + +if __name__ == "__main__": + """ + this is a standalone script to process a km file + specifically, to dedup or remove tokens that repeat less + than k times in a row + """ + parser = argparse.ArgumentParser(description="") + parser.add_argument("km", type=str, help="path to km file") + parser.add_argument("--dedup", action='store_true') + parser.add_argument("--remove-under-k", type=int, default=0) + parser.add_argument("--output", default=None) + args = parser.parse_args() + + if not args.dedup and args.remove_under_k == 0: + print("nothing to do! quitting...") + sys.exit(0) + + km = open(args.km, "r").readlines() + out = [] + for line in tqdm(km): + if args.remove_under_k > 0: + line = remove_under_k(line, args.remove_under_k) + if args.dedup: + line = dedup(line) + out.append(line) + + path = args.km if args.output is None else args.output + if args.remove_under_k > 0: + path = path.replace(".km", f"-k{args.remove_under_k}.km") + if args.dedup: + path = path.replace(".km", f"-deduped.km") + + open(path, "w").writelines(out) + print(f"written to {path}") diff --git a/data/fairseq/examples/emotion_conversion/preprocess/split_emov_km_tsv_by_uttid.py b/data/fairseq/examples/emotion_conversion/preprocess/split_emov_km_tsv_by_uttid.py new file mode 100644 index 0000000000000000000000000000000000000000..94221afba711eb106ecd2275e5862a9a8b6148fd --- /dev/null +++ b/data/fairseq/examples/emotion_conversion/preprocess/split_emov_km_tsv_by_uttid.py @@ -0,0 +1,70 @@ +from pathlib import Path +import os +import sys +import argparse +import random +import numpy as np +from tqdm import tqdm +from sklearn.model_selection import train_test_split +from build_translation_manifests import get_utt_id + + +def train_val_test_split(tsv_lines, km_lines, valid_percent, test_percent, seed=42): + utt_ids = list(sorted(set([get_utt_id(x) for x in tsv_lines]))) + utt_ids, valid_utt_ids, _, _ = train_test_split(utt_ids, utt_ids, test_size=valid_percent, shuffle=True, random_state=seed) + train_utt_ids, test_utt_ids, _, _ = train_test_split(utt_ids, utt_ids, test_size=test_percent, shuffle=True, random_state=seed) + + train_idx = [i for i, line in enumerate(tsv_lines) if get_utt_id(line) in train_utt_ids] + valid_idx = [i for i, line in enumerate(tsv_lines) if get_utt_id(line) in valid_utt_ids] + test_idx = [i for i, line in enumerate(tsv_lines) if get_utt_id(line) in test_utt_ids] + + train_tsv, train_km = [tsv_lines[i] for i in train_idx], [km_lines[i] for i in train_idx] + valid_tsv, valid_km = [tsv_lines[i] for i in valid_idx], [km_lines[i] for i in valid_idx] + test_tsv, test_km = [tsv_lines[i] for i in test_idx], [km_lines[i] for i in test_idx] + + print(f"train {len(train_km)}") + print(f"valid {len(valid_km)}") + print(f"test {len(test_km)}") + + return train_tsv, train_km, valid_tsv, valid_km, test_tsv, test_km + + +if __name__ == "__main__": + """ + this is a standalone script to process a km file + specifically, to dedup or remove tokens that repeat less + than k times in a row + """ + parser = argparse.ArgumentParser(description="") + parser.add_argument("tsv", type=str, help="path to tsv file") + parser.add_argument("km", type=str, help="path to km file") + parser.add_argument("--destdir", required=True, type=str) + parser.add_argument("--valid-percent", type=float, default=0.05, help="percent to allocate to validation set") + parser.add_argument("--test-percent", type=float, default=0.05, help="percent to allocate to test set") + parser.add_argument("--seed", type=int, default=42, help="") + args = parser.parse_args() + + np.random.seed(args.seed) + random.seed(args.seed) + + os.makedirs(args.destdir, exist_ok=True) + km = open(args.km, "r").readlines() + tsv = open(args.tsv, "r").readlines() + root, tsv = tsv[0], tsv[1:] + + assert args.tsv.endswith(".tsv") and args.km.endswith(".km") + assert len(tsv) == len(km) + + train_tsv, train_km, valid_tsv, valid_km, test_tsv, test_km = train_val_test_split(tsv, km, args.valid_percent, args.test_percent, args.seed) + + assert len(train_tsv) + len(valid_tsv) + len(test_tsv) == len(tsv) + assert len(train_tsv) == len(train_km) and len(valid_tsv) == len(valid_km) and len(test_tsv) == len(test_km) + + dir = Path(args.destdir) + open(dir / f"train.tsv", "w").writelines([root] + train_tsv) + open(dir / f"valid.tsv", "w").writelines([root] + valid_tsv) + open(dir / f"test.tsv", "w").writelines([root] + test_tsv) + open(dir / f"train.km", "w").writelines(train_km) + open(dir / f"valid.km", "w").writelines(valid_km) + open(dir / f"test.km", "w").writelines(test_km) + print("done") diff --git a/data/fairseq/examples/emotion_conversion/preprocess/split_km.py b/data/fairseq/examples/emotion_conversion/preprocess/split_km.py new file mode 100644 index 0000000000000000000000000000000000000000..d145fc2bdefe72d5ce90b56f13d4e9f19a001d31 --- /dev/null +++ b/data/fairseq/examples/emotion_conversion/preprocess/split_km.py @@ -0,0 +1,50 @@ +from pathlib import Path +import os +import argparse +import random +import numpy as np +from sklearn.utils import shuffle + + +if __name__ == "__main__": + """ + this is a standalone script to process a km file + specifically, to dedup or remove tokens that repeat less + than k times in a row + """ + parser = argparse.ArgumentParser(description="") + parser.add_argument("km", type=str, help="path to km file") + parser.add_argument("--destdir", required=True, type=str) + parser.add_argument("--valid-percent", type=float, default=0.05, help="percent to allocate to validation set") + parser.add_argument("--test-percent", type=float, default=0.05, help="percent to allocate to test set") + parser.add_argument("-sh", "--shuffle", action="store_true", help="path to km file") + parser.add_argument("--seed", type=int, default=42, help="") + args = parser.parse_args() + + np.random.seed(args.seed) + random.seed(args.seed) + + os.makedirs(args.destdir, exist_ok=True) + km = open(args.km, "r").readlines() + + if args.shuffle: + km = shuffle(km) + print(f"shuffled") + + N = len(km) + N_tt = int(N * args.test_percent) + N_cv = int(N * args.valid_percent) + N_tr = N - N_tt - N_cv + + train_km = km[:N_tr] + valid_km = km[N_tr:N_tr + N_cv] + test_km = km[N_tr + N_cv:] + + dir = Path(args.destdir) + open(dir / f"train.km", "w").writelines(train_km) + open(dir / f"valid.km", "w").writelines(valid_km) + open(dir / f"test.km", "w").writelines(test_km) + print(f"train: {len(train_km)}") + print(f"valid: {len(valid_km)}") + print(f"test: {len(test_km)}") + print("done") diff --git a/data/fairseq/examples/emotion_conversion/preprocess/split_km_tsv.py b/data/fairseq/examples/emotion_conversion/preprocess/split_km_tsv.py new file mode 100644 index 0000000000000000000000000000000000000000..2113aa718d3bb75c08e5f9d01a79c2ff7ff2d4f4 --- /dev/null +++ b/data/fairseq/examples/emotion_conversion/preprocess/split_km_tsv.py @@ -0,0 +1,65 @@ +from pathlib import Path +import os +import argparse +import random +import numpy as np +from sklearn.utils import shuffle + + +if __name__ == "__main__": + """ + this is a standalone script to process a km file + specifically, to dedup or remove tokens that repeat less + than k times in a row + """ + parser = argparse.ArgumentParser(description="") + parser.add_argument("tsv", type=str, help="path to tsv file") + parser.add_argument("km", type=str, help="path to km file") + parser.add_argument("--destdir", required=True, type=str) + parser.add_argument("--valid-percent", type=float, default=0.05, help="percent to allocate to validation set") + parser.add_argument("--test-percent", type=float, default=0.05, help="percent to allocate to test set") + parser.add_argument("-sh", "--shuffle", action="store_true", help="path to km file") + parser.add_argument("--seed", type=int, default=42, help="") + args = parser.parse_args() + + np.random.seed(args.seed) + random.seed(args.seed) + + os.makedirs(args.destdir, exist_ok=True) + km = open(args.km, "r").readlines() + tsv = open(args.tsv, "r").readlines() + root, tsv = tsv[0], tsv[1:] + + assert args.tsv.endswith(".tsv") and args.km.endswith(".km") + assert len(tsv) == len(km) + + if args.shuffle: + tsv, km = shuffle(tsv, km) + print(f"shuffled") + + N = len(tsv) + N_tt = int(N * args.test_percent) + N_cv = int(N * args.valid_percent) + N_tr = N - N_tt - N_cv + + train_tsv = tsv[:N_tr] + valid_tsv = tsv[N_tr:N_tr + N_cv] + test_tsv = tsv[N_tr + N_cv:] + train_km = km[:N_tr] + valid_km = km[N_tr:N_tr + N_cv] + test_km = km[N_tr + N_cv:] + + assert len(train_tsv) + len(valid_tsv) + len(test_tsv) == len(tsv) + assert len(train_tsv) == len(train_km) and len(valid_tsv) == len(valid_km) and len(test_tsv) == len(test_km) + + dir = Path(args.destdir) + open(dir / f"train.tsv", "w").writelines([root] + train_tsv) + open(dir / f"valid.tsv", "w").writelines([root] + valid_tsv) + open(dir / f"test.tsv", "w").writelines([root] + test_tsv) + open(dir / f"train.km", "w").writelines(train_km) + open(dir / f"valid.km", "w").writelines(valid_km) + open(dir / f"test.km", "w").writelines(test_km) + print(f"train: {len(train_km)}") + print(f"valid: {len(valid_km)}") + print(f"test: {len(test_km)}") + print("done") diff --git a/data/fairseq/examples/emotion_conversion/requirements.txt b/data/fairseq/examples/emotion_conversion/requirements.txt new file mode 100644 index 0000000000000000000000000000000000000000..fc94c5a5470833d45468745a78fea1844e746dbb --- /dev/null +++ b/data/fairseq/examples/emotion_conversion/requirements.txt @@ -0,0 +1,11 @@ +scipy +einops +amfm_decompy +joblib +numba +decorator +requests +appdirs +packaging +six +sklearn diff --git a/data/fairseq/examples/emotion_conversion/synthesize.py b/data/fairseq/examples/emotion_conversion/synthesize.py new file mode 100644 index 0000000000000000000000000000000000000000..327fdaf4ea38a08d63cfb2d9b9a18d27e8e326c7 --- /dev/null +++ b/data/fairseq/examples/emotion_conversion/synthesize.py @@ -0,0 +1,322 @@ +import logging +import argparse +import random +import sys +import os +import numpy as np +import torch +import soundfile as sf +import shutil +import librosa +import json +from pathlib import Path +from tqdm import tqdm +import amfm_decompy.basic_tools as basic +import amfm_decompy.pYAAPT as pYAAPT + +dir_path = os.path.dirname(__file__) +resynth_path = os.path.dirname(os.path.abspath(__file__)) + "/speech-resynthesis" +sys.path.append(resynth_path) + +from models import CodeGenerator +from inference import scan_checkpoint, load_checkpoint, generate +from emotion_models.pitch_predictor import load_ckpt as load_pitch_predictor +from emotion_models.duration_predictor import load_ckpt as load_duration_predictor +from dataset import load_audio, MAX_WAV_VALUE, parse_style, parse_speaker, EMOV_SPK2ID, EMOV_STYLE2ID + + +logging.basicConfig( + level=logging.INFO, + format='%(asctime)s [%(levelname)s] %(message)s', + handlers=[logging.FileHandler('debug.log'), logging.StreamHandler()] +) +logger = logging.getLogger(__name__) + + +class AttrDict(dict): + def __init__(self, *args, **kwargs): + super(AttrDict, self).__init__(*args, **kwargs) + self.__dict__ = self + + +def parse_generation_file(fname): + lines = open(fname).read() + lines = lines.split('\n') + + results = {} + for l in lines: + if len(l) == 0: + continue + + if l[0] == 'H': + parts = l[2:].split('\t') + if len(parts) == 2: + sid, utt = parts + else: + sid, _, utt = parts + sid = int(sid) + utt = [int(x) for x in utt.split()] + if sid in results: + results[sid]['H'] = utt + else: + results[sid] = {'H': utt} + elif l[0] == 'S': + sid, utt = l[2:].split('\t') + sid = int(sid) + utt = [x for x in utt.split()] + if sid in results: + results[sid]['S'] = utt + else: + results[sid] = {'S': utt} + elif l[0] == 'T': + sid, utt = l[2:].split('\t') + sid = int(sid) + utt = [int(x) for x in utt.split()] + if sid in results: + results[sid]['T'] = utt + else: + results[sid] = {'T': utt} + + for d, result in results.items(): + if 'H' not in result: + result['H'] = result['S'] + + return results + + +def get_code_to_fname(manifest, tokens): + if tokens is None: + code_to_fname = {} + with open(manifest) as f: + for line in f: + line = line.strip() + fname, code = line.split() + code = code.replace(',', ' ') + code_to_fname[code] = fname + + return code_to_fname + + with open(manifest) as f: + fnames = [l.strip() for l in f.readlines()] + root = Path(fnames[0]) + fnames = fnames[1:] + if '\t' in fnames[0]: + fnames = [x.split()[0] for x in fnames] + + with open(tokens) as f: + codes = [l.strip() for l in f.readlines()] + + code_to_fname = {} + for fname, code in zip(fnames, codes): + code = code.replace(',', ' ') + code_to_fname[code] = str(root / fname) + + return root, code_to_fname + + +def code_to_str(s): + k = ' '.join([str(x) for x in s]) + return k + + +def get_praat_f0(audio, rate=16000, interp=False): + frame_length = 20.0 + to_pad = int(frame_length / 1000 * rate) // 2 + + f0s = [] + for y in audio.astype(np.float64): + y_pad = np.pad(y.squeeze(), (to_pad, to_pad), "constant", constant_values=0) + signal = basic.SignalObj(y_pad, rate) + pitch = pYAAPT.yaapt(signal, **{'frame_length': frame_length, 'frame_space': 5.0, 'nccf_thresh1': 0.25, + 'tda_frame_length': 25.0}) + if interp: + f0s += [pitch.samp_interp[None, None, :]] + else: + f0s += [pitch.samp_values[None, None, :]] + + f0 = np.vstack(f0s) + return f0 + + +def generate_from_code(generator, h, code, spkr=None, f0=None, gst=None, device="cpu"): + batch = { + 'code': torch.LongTensor(code).to(device).view(1, -1), + } + if spkr is not None: + batch['spkr'] = spkr.to(device).unsqueeze(0) + if f0 is not None: + batch['f0'] = f0.to(device) + if gst is not None: + batch['style'] = gst.to(device) + + with torch.no_grad(): + audio, rtf = generate(h, generator, batch) + audio = librosa.util.normalize(audio / 2 ** 15) + + return audio + + +@torch.no_grad() +def synth(argv, interactive=False): + parser = argparse.ArgumentParser() + parser.add_argument('--result-path', type=Path, help='Translation Model Output', required=True) + parser.add_argument('--data', type=Path, help='a directory with the files: src.tsv, src.km, trg.tsv, trg.km, orig.tsv, orig.km') + parser.add_argument("--orig-tsv", default="/checkpoint/felixkreuk/datasets/emov/manifests/emov_16khz/data.tsv") + parser.add_argument("--orig-km", default="/checkpoint/felixkreuk/datasets/emov/manifests/emov_16khz/core_manifests/emov_16khz_km_100/data.km") + + parser.add_argument('--checkpoint-file', type=Path, help='Generator Checkpoint', required=True) + parser.add_argument('--dur-model', type=Path, help='a token duration prediction model (if tokens were deduped)') + parser.add_argument('--f0-model', type=Path, help='a f0 prediction model') + + parser.add_argument('-s', '--src-emotion', default=None) + parser.add_argument('-t', '--trg-emotion', default=None) + parser.add_argument('-N', type=int, default=10) + parser.add_argument('--split', default="test") + + parser.add_argument('--outdir', type=Path, default=Path('results')) + parser.add_argument('--orig-filename', action='store_true') + + parser.add_argument('--device', type=int, default=0) + a = parser.parse_args(argv) + + seed = 52 + random.seed(seed) + np.random.seed(seed) + torch.manual_seed(seed) + + if os.path.isdir(a.checkpoint_file): + config_file = os.path.join(a.checkpoint_file, 'config.json') + else: + config_file = os.path.join(os.path.split(a.checkpoint_file)[0], 'config.json') + with open(config_file) as f: + data = f.read() + json_config = json.loads(data) + h = AttrDict(json_config) + + generator = CodeGenerator(h).to(a.device) + if os.path.isdir(a.checkpoint_file): + cp_g = scan_checkpoint(a.checkpoint_file, 'g_') + else: + cp_g = a.checkpoint_file + state_dict_g = load_checkpoint(cp_g) + generator.load_state_dict(state_dict_g['generator']) + + generator.eval() + generator.remove_weight_norm() + + dur_models = { + "neutral": load_duration_predictor(f"{a.dur_model}/neutral.ckpt"), + "amused": load_duration_predictor(f"{a.dur_model}/amused.ckpt"), + "disgusted": load_duration_predictor(f"{a.dur_model}/disgusted.ckpt"), + "angry": load_duration_predictor(f"{a.dur_model}/angry.ckpt"), + "sleepy": load_duration_predictor(f"{a.dur_model}/sleepy.ckpt"), + } + logger.info(f"loaded duration prediction model from {a.dur_model}") + + f0_model = load_pitch_predictor(a.f0_model).to(a.device) + logger.info(f"loaded f0 prediction model from {a.f0_model}") + + # we need to know how to map code back to the filename + # (if we want the original files names as output) + results = parse_generation_file(a.result_path) + _, src_code_to_fname = get_code_to_fname(f'{a.data}/files.{a.split}.{a.src_emotion}', f'{a.data}/{a.split}.{a.src_emotion}') + _, tgt_code_to_fname = get_code_to_fname(f'{a.data}/files.{a.split}.{a.trg_emotion}', f'{a.data}/{a.split}.{a.trg_emotion}') + + # we need the originals (before dedup) to get the ground-truth durations + orig_tsv = open(a.orig_tsv, 'r').readlines() + orig_tsv_root, orig_tsv = orig_tsv[0].strip(), orig_tsv[1:] + orig_km = open(a.orig_km, 'r').readlines() + fname_to_idx = {orig_tsv_root + "/" + line.split("\t")[0]: i for i, line in enumerate(orig_tsv)} + + outdir = a.outdir + outdir.mkdir(parents=True, exist_ok=True) + (outdir / '0-source').mkdir(exist_ok=True) + (outdir / '1-src-tokens-src-style-src-f0').mkdir(exist_ok=True) + (outdir / '2-src-tokens-trg-style-src-f0').mkdir(exist_ok=True) + (outdir / '2.5-src-tokens-trg-style-src-f0').mkdir(exist_ok=True) + (outdir / '3-src-tokens-trg-style-pred-f0').mkdir(exist_ok=True) + (outdir / '4-gen-tokens-trg-style-pred-f0').mkdir(exist_ok=True) + (outdir / '5-target').mkdir(exist_ok=True) + + N = 0 + results = list(results.items()) + random.shuffle(results) + for i, (sid, result) in tqdm(enumerate(results)): + N += 1 + if N > a.N and a.N != -1: + break + + if '[' in result['S'][0]: + result['S'] = result['S'][1:] + if '_' in result['S'][-1]: + result['S'] = result['S'][:-1] + src_ref = src_code_to_fname[code_to_str(result['S'])] + trg_ref = tgt_code_to_fname[code_to_str(result['T'])] + + src_style, trg_style = None, None + src_spkr, trg_spkr = None, None + src_f0 = None + src_audio = (load_audio(src_ref)[0] / MAX_WAV_VALUE) * 0.95 + trg_audio = (load_audio(trg_ref)[0] / MAX_WAV_VALUE) * 0.95 + src_audio = torch.FloatTensor(src_audio).unsqueeze(0).cuda() + trg_audio = torch.FloatTensor(trg_audio).unsqueeze(0).cuda() + + src_spkr = parse_speaker(src_ref, h.multispkr) + src_spkr = src_spkr if src_spkr in EMOV_SPK2ID else random.choice(list(EMOV_SPK2ID.keys())) + src_spkr = EMOV_SPK2ID[src_spkr] + src_spkr = torch.LongTensor([src_spkr]) + trg_spkr = parse_speaker(trg_ref, h.multispkr) + trg_spkr = trg_spkr if trg_spkr in EMOV_SPK2ID else random.choice(list(EMOV_SPK2ID.keys())) + trg_spkr = EMOV_SPK2ID[trg_spkr] + trg_spkr = torch.LongTensor([trg_spkr]) + + src_style = EMOV_STYLE2ID[a.src_emotion] + src_style = torch.LongTensor([src_style]).cuda() + trg_style_str = a.trg_emotion + trg_style = EMOV_STYLE2ID[a.trg_emotion] + trg_style = torch.LongTensor([trg_style]).cuda() + + src_tokens = list(map(int, orig_km[fname_to_idx[src_ref]].strip().split(" "))) + src_tokens = torch.LongTensor(src_tokens).unsqueeze(0) + src_tokens_dur_pred = torch.LongTensor(list(map(int, result['S']))).unsqueeze(0) + src_tokens_dur_pred = dur_models[trg_style_str].inflate_input(src_tokens_dur_pred) + gen_tokens = torch.LongTensor(result['H']).unsqueeze(0) + gen_tokens = dur_models[trg_style_str].inflate_input(gen_tokens) + trg_tokens = torch.LongTensor(result['T']).unsqueeze(0) + trg_tokens = dur_models[trg_style_str].inflate_input(trg_tokens) + + src_f0 = get_praat_f0(src_audio.unsqueeze(0).cpu().numpy()) + src_f0 = torch.FloatTensor(src_f0).cuda() + + pred_src_f0 = f0_model.inference(torch.LongTensor(src_tokens).to(a.device), src_spkr, trg_style).unsqueeze(0) + pred_src_dur_pred_f0 = f0_model.inference(torch.LongTensor(src_tokens_dur_pred).to(a.device), src_spkr, trg_style).unsqueeze(0) + pred_gen_f0 = f0_model.inference(torch.LongTensor(gen_tokens).to(a.device), src_spkr, trg_style).unsqueeze(0) + pred_trg_f0 = f0_model.inference(torch.LongTensor(trg_tokens).to(a.device), src_spkr, trg_style).unsqueeze(0) + + if a.orig_filename: + path = src_code_to_fname[code_to_str(result['S'])] + sid = str(sid) + "__" + Path(path).stem + shutil.copy(src_code_to_fname[code_to_str(result['S'])], outdir / '0-source' / f'{sid}.wav') + + audio = generate_from_code(generator, h, src_tokens, spkr=src_spkr, f0=src_f0, gst=src_style, device=a.device) + sf.write(outdir / '1-src-tokens-src-style-src-f0' / f'{sid}.wav', audio, samplerate=h.sampling_rate) + + audio = generate_from_code(generator, h, src_tokens, spkr=src_spkr, f0=src_f0, gst=trg_style, device=a.device) + sf.write(outdir / '2-src-tokens-trg-style-src-f0' / f'{sid}.wav', audio, samplerate=h.sampling_rate) + + audio = generate_from_code(generator, h, src_tokens_dur_pred, spkr=src_spkr, f0=src_f0, gst=trg_style, device=a.device) + sf.write(outdir / '2.5-src-tokens-trg-style-src-f0' / f'{sid}.wav', audio, samplerate=h.sampling_rate) + + audio = generate_from_code(generator, h, src_tokens_dur_pred, spkr=src_spkr, f0=pred_src_dur_pred_f0, gst=trg_style, device=a.device) + sf.write(outdir / '3-src-tokens-trg-style-pred-f0' / f'{sid}.wav', audio, samplerate=h.sampling_rate) + + audio = generate_from_code(generator, h, gen_tokens, spkr=src_spkr, f0=pred_gen_f0, gst=trg_style, device=a.device) + sf.write(outdir / '4-gen-tokens-trg-style-pred-f0' / f'{sid}.wav', audio, samplerate=h.sampling_rate) + + shutil.copy(tgt_code_to_fname[code_to_str(result['T'])], outdir / '5-target' / f'{sid}.wav') + + logger.info("Done.") + + +if __name__ == '__main__': + synth(sys.argv[1:]) diff --git a/data/fairseq/examples/fast_noisy_channel/README.md b/data/fairseq/examples/fast_noisy_channel/README.md new file mode 100644 index 0000000000000000000000000000000000000000..f2631a8c34d11bdf7d351c6807b6fe415f5715e1 --- /dev/null +++ b/data/fairseq/examples/fast_noisy_channel/README.md @@ -0,0 +1,345 @@ +# Language Models not just for Pre-training: Fast Online Neural Noisy Channel Modeling + +## Introduction +- [Yee et al. (2019)](https://www.aclweb.org/anthology/D19-1571.pdf) introduce a simple and effective noisy channel modeling approach for neural machine translation. However, the noisy channel online decoding approach introduced in this paper is too slow to be practical. +- To address this, [Bhosale et al. (2020)](http://www.statmt.org/wmt20/pdf/2020.wmt-1.68.pdf) introduces 3 simple approximations to make this approach very fast and practical without much loss in accuracy. +- This README provides intructions on how to run online decoding or generation with the noisy channel modeling approach, including ways to make it very fast without much loss in accuracy. + +## Noisy Channel Modeling + +[Yee et al. (2019)](https://www.aclweb.org/anthology/D19-1571.pdf) applies the Bayes Rule to predict `P(y|x)`, the probability of the target `y` given the source `x`. +```P(y|x) = P(x|y) * P(y) / P(x)``` +- `P(x|y)` predicts the source `x` given the target `y` and is referred to as the **channel model** +- `P(y)` is a **language model** over the target `y` +- `P(x)` is generally not modeled since it is constant for all `y`. + +We use Transformer models to parameterize the direct model `P(y|x)`, the channel model `P(x|y)` and the language model `P(y)`. + +During online decoding with beam search, we generate the top `K2` candidates per beam and score them with the following linear combination of the channel model, the language model as well as the direct model scores. + +```(1 / t) * log(P(y|x) + (1 / s) * ( λ1 * log(P(x|y)) + λ2 * log(P(y) ) )``` +- `t` - Target Prefix Length +- `s` - Source Length +- `λ1` - Channel Model Weight +- `λ2` - Language Model Weight + +The top `beam_size` candidates based on the above combined scores are chosen to continue the beams in beam search. In beam search with a direct model alone, the scores from the direct model `P(y|x)` are used to choose the top candidates in beam search. + +This framework provides a great way to utlize strong target language models trained on large amounts of unlabeled data. Language models can prefer targets unrelated to the source, so we also need a channel model whose role is to ensure that the target preferred by the language model also translates back to the source. + +### Training Translation Models and Language Models + +For training Transformer models in fairseq for machine translation, refer to instructions [here](https://github.com/pytorch/fairseq/tree/main/examples/translation) + +For training Transformer models in fairseq for language modeling, refer to instructions [here](https://github.com/pytorch/fairseq/tree/main/examples/language_model) + +### Generation with Language Model for German-English translation with fairseq + +Here are instructions to generate using a direct model and a target-side language model. + +Note: +- Download and install fairseq as per instructions [here](https://github.com/pytorch/fairseq) +- Preprocess and binarize the dataset as per instructions in section [Test Data Preprocessing](#test-data-preprocessing) + +```sh +binarized_data=data_dir/binarized +direct_model=de_en_seed4.pt +lm_model=en_lm.pt +lm_data=lm_data +wget https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/direct_models/seed4.pt -O ${direct_model} +wget https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/lm_model/transformer_lm.pt -O ${lm_model} +mkdir -p ${lm_data} +wget https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/lm_model/lm_dict/dict.txt -O ${lm_data}/dict.txt + +k2=10 +lenpen=0.16 +lm_wt=0.14 +fairseq-generate ${binarized_data} \ + --user-dir examples/fast_noisy_channel \ + --beam 5 \ + --path ${direct_model} \ + --lm-model ${lm_model} \ + --lm-data ${lm_data} \ + --k2 ${k2} \ + --combine-method lm_only \ + --task noisy_channel_translation \ + --lenpen ${lenpen} \ + --lm-wt ${lm_wt} \ + --gen-subset valid \ + --remove-bpe \ + --fp16 \ + --batch-size 10 +``` +### Noisy Channel Generation for German-English translation with fairseq + +Here are instructions for noisy channel generation with a direct model, channel model and language model as explained in section [Noisy Channel Modeling](#noisy-channel-modeling). + +Note: +- Download and install fairseq as per instructions [here](https://github.com/pytorch/fairseq) +- Preprocess and binarize the dataset as per instructions in section [Test Data Preprocessing](#test-data-preprocessing) + +```sh +binarized_data=data_dir/binarized +direct_model=de_en_seed4.pt +lm_model=en_lm.pt +lm_data=lm_data +ch_model=en_de.big.seed4.pt +wget https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/direct_models/seed4.pt -O ${direct_model} +wget https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/lm_model/transformer_lm.pt -O ${lm_model} +mkdir -p ${lm_data} +wget https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/lm_model/lm_dict/dict.txt -O ${lm_data}/dict.txt +wget https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/big.seed4.pt -O ${ch_model} + +k2=10 +lenpen=0.21 +lm_wt=0.50 +bw_wt=0.30 +fairseq-generate ${binarized_data} \ + --user-dir examples/fast_noisy_channel \ + --beam 5 \ + --path ${direct_model} \ + --lm-model ${lm_model} \ + --lm-data ${lm_data} \ + --channel-model ${ch_model} \ + --k2 ${k2} \ + --combine-method noisy_channel \ + --task noisy_channel_translation \ + --lenpen ${lenpen} \ + --lm-wt ${lm_wt} \ + --ch-wt ${bw_wt} \ + --gen-subset test \ + --remove-bpe \ + --fp16 \ + --batch-size 1 +``` +## Fast Noisy Channel Modeling + +[Bhosale et al. (2020)](http://www.statmt.org/wmt20/pdf/2020.wmt-1.68.pdf) introduces 3 approximations that speed up online noisy channel decoding - +- Smaller channel models (`Tranformer Base` with 1 encoder and decoder layer each vs. `Transformer Big`) + - This involves training a channel model that is possibly smaller and less accurate in terms of BLEU than a channel model of the same size as the direct model. + - Since the role of the channel model is mainly to assign low scores to generations from the language model if they don't translate back to the source, we may not need the most accurate channel model for this purpose. +- Smaller output vocabulary size for the channel model (~30,000 -> ~1000) + - The channel model doesn't need to score the full output vocabulary, it just needs to score the source tokens, which are completely known. + - This is specified using the arguments `--channel-scoring-type src_vocab --top-k-vocab 500` + - This means that the output vocabulary for the channel model will be the source tokens for all examples in the batch and the top-K most frequent tokens in the vocabulary + - This reduces the memory consumption needed to store channel model scores significantly +- Smaller number of candidates (`k2`) scored per beam + - This is specified by reducing the argument `--k2` + + +### Fast Noisy Channel Generation for German-English translation with fairseq + +Here are instructions for **fast** noisy channel generation with a direct model, channel model and language model as explained in section [Fast Noisy Channel Modeling](#fast-noisy-channel-modeling). The main differences are that we use a smaller channel model, reduce `--k2`, set `--channel-scoring-type src_vocab --top-k-vocab 500` and increase the `--batch-size`. + +Note: +- Download and install fairseq as per instructions [here](https://github.com/pytorch/fairseq) +- Preprocess and binarize the dataset as per instructions in section [Test Data Preprocessing](#test-data-preprocessing) + +```sh +binarized_data=data_dir/binarized +direct_model=de_en_seed4.pt +lm_model=en_lm.pt +lm_data=lm_data +small_ch_model=en_de.base_1_1.seed4.pt +wget https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/direct_models/seed4.pt -O ${direct_model} +wget https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/lm_model/transformer_lm.pt -O ${lm_model} +mkdir -p ${lm_data} +wget https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/lm_model/lm_dict/dict.txt -O ${lm_data}/dict.txt +wget https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/base_1_1.seed4.pt -O ${small_ch_model} + +k2=3 +lenpen=0.23 +lm_wt=0.58 +bw_wt=0.26 +fairseq-generate ${binarized_data} \ + --user-dir examples/fast_noisy_channel \ + --beam 5 \ + --path ${direct_model} \ + --lm-model ${lm_model} \ + --lm-data ${lm_data} \ + --channel-model ${small_ch_model} \ + --k2 ${k2} \ + --combine-method noisy_channel \ + --task noisy_channel_translation \ + --lenpen ${lenpen} \ + --lm-wt ${lm_wt} \ + --ch-wt ${bw_wt} \ + --gen-subset test \ + --remove-bpe \ + --fp16 \ + --batch-size 50 \ + --channel-scoring-type src_vocab --top-k-vocab 500 +``` + +## Test Data Preprocessing + +For preprocessing and binarizing the test sets for Romanian-English and German-English translation, we use the following script - + +```sh +FAIRSEQ=/path/to/fairseq +cd $FAIRSEQ +SCRIPTS=$FAIRSEQ/mosesdecoder/scripts +if [ ! -d "${SCRIPTS}" ]; then + echo 'Cloning Moses github repository (for tokenization scripts)...' + git clone https://github.com/moses-smt/mosesdecoder.git +fi +TOKENIZER=$SCRIPTS/tokenizer/tokenizer.perl +NORMALIZE=$SCRIPTS/tokenizer/normalize-punctuation.perl + +s=de +t=en +test=wmt18 + +mkdir -p data_dir + +# Tokenization +if [ $s == "ro" ] ; then + # Note: Get normalise-romanian.py and remove-diacritics.py from + # https://github.com/rsennrich/wmt16-scripts/tree/master/preprocess + sacrebleu -t $test -l $s-$t --echo src | \ + $NORMALIZE -l $s | \ + python normalise-romanian.py | \ + python remove-diacritics.py | \ + $TOKENIZER -l $s -a -q > data_dir/$test.$s-$t.$s +else + sacrebleu -t $test -l $s-$t --echo src | perl $NORMALIZE -l $s | perl $TOKENIZER -threads 8 -a -l $s > data_dir/$test.$s-$t.$s +fi + +sacrebleu -t $test -l $s-$t --echo ref | perl $NORMALIZE -l $t | perl $TOKENIZER -threads 8 -a -l $t > data_dir/$test.$s-$t.$t + + +# Applying BPE +src_bpe_code=/path/to/source/language/bpe/code +tgt_bpe_code=/path/to/target/language/bpe/code +src_dict=/path/to/source/language/dict +tgt_dict=/path/to/target/language/dict + +FASTBPE=$FAIRSEQ/fastBPE +if [ ! -d "${FASTBPE}" ] ; then + git clone https://github.com/glample/fastBPE.git + # Follow compilation instructions at https://github.com/glample/fastBPE + g++ -std=c++11 -pthread -O3 fastBPE/main.cc -IfastBPE -o fast +fi + +${FASTBPE}/fast applybpe data_dir/bpe.$test.$s-$t.$s data_dir/$test.$s-$t.$s ${src_bpe_code} +${FASTBPE}/fast applybpe data_dir/bpe.$test.$s-$t.$s data_dir/$test.$s-$t.$s ${tgt_bpe_code} + +fairseq-preprocess -s $s -t $t \ + --testpref data_dir/bpe.$test.$s-$t \ + --destdir data_dir/binarized \ + --srcdict ${src_dict} \ + --tgtdict ${tgt_dict} +``` + +## Calculating BLEU + +```sh +DETOKENIZER=$SCRIPTS/tokenizer/detokenizer.perl +cat ${generation_output} | grep -P "^H" | sort -V | cut -f 3- | $DETOKENIZER -l $t -q -a | sacrebleu -t $test -l $s-$t +``` + + +## Romanian-English Translation + +The direct and channel models are trained using bitext data (WMT16) combined with backtranslated data (The monolingual data used for backtranslation comes from http://data.statmt.org/rsennrich/wmt16_backtranslations/ (Sennrich et al., 2016c)) + +The backtranslated data is generated using an ensemble of 3 English-Romanian models trained on bitext training data (WMT16) with unrestricted sampling. + +### BPE Codes and Dictionary + +We learn a joint BPE vocabulary of 18K types on the bitext training data which is used for both the source and target. +||Path| +|----------|------| +| BPE Code | [joint_bpe_18k](https://dl.fbaipublicfiles.com/fast_noisy_channel/ro_en/bpe_18k) | +| Dictionary | [dict](https://dl.fbaipublicfiles.com/fast_noisy_channel/ro_en/dict) | + +### Direct Models +For Ro-En with backtranslation, the direct and channel models use a Transformer-Big architecture. + +| Seed | Model | +|----|----| +| 2 | [ro_en_seed2.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/ro_en/direct_models/seed2.pt) +| 4 | [ro_en_seed4.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/ro_en/direct_models/seed4.pt) +| 6 | [ro_en_seed6.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/ro_en/direct_models/seed6.pt) + +### Channel Models +For channel models, we follow the same steps as for the direct models. But backtranslated data is generated in the opposite direction using [this Romanian monolingual data](http://data.statmt.org/rsennrich/wmt16_backtranslations/). +The best lenpen, LM weight and CH weight are obtained by sweeping over the validation set (wmt16/dev) using beam 5. +| Model Size | Lenpen | LM Weight | CH Weight | Seed 2 | Seed 4 | Seed 6 | +|----|----|----|----|----|----|----| +| `big` | 0.84 | 0.64 | 0.56 | [big.seed2.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/ro_en/channel_models/big.seed2.pt) | [big.seed2.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/ro_en/channel_models/big.seed2.pt) | [big.seed2.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/ro_en/channel_models/big.seed2.pt) | +| `base_1_1` | 0.63 | 0.40 | 0.37 | [base_1_1.seed2.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/ro_en/channel_models/base_1_1.seed2.pt) | [base_1_1.seed4.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/ro_en/channel_models/base_1_1.seed4.pt) | [base_1_1.seed6.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/ro_en/channel_models/base_1_1.seed6.pt) | + +### Language Model +The model is trained on de-duplicated English Newscrawl data from 2007-2018 comprising 186 million sentences or 4.5B words after normalization and tokenization. +| | Path | +|----|----| +| `--lm-model` | [transformer_en_lm](https://dl.fbaipublicfiles.com/fast_noisy_channel/ro_en/lm_model/transformer_lm.pt) | +| `--lm-data` | [lm_data](https://dl.fbaipublicfiles.com/fast_noisy_channel/ro_en/lm_model/lm_dict) + +## German-English Translation + +### BPE Codes and Dictionaries + +| | Path| +|----------|------| +| Source BPE Code | [de_bpe_code_24K](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/de_bpe_code_24K) | +| Target BPE Code | [en_bpe_code_24K](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/en_bpe_code_24K) +| Source Dictionary | [de_dict](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/de_dict) | +| Target Dictionary | [en_dict](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/en_dict) | + +### Direct Models +We train on WMT’19 training data. Following [Ng et al., 2019](http://statmt.org/wmt19/pdf/53/WMT33.pdf), we apply language identification filtering and remove sentences longer than 250 tokens as well as sentence pairs with a source/target length ratio exceeding 1.5. This results in 26.8M sentence pairs. +We use the Transformer-Big architecture for the direct model. + +| Seed | Model | +|:----:|----| +| 4 | [de_en_seed4.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/direct_models/seed4.pt) +| 5 | [de_en_seed5.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/direct_models/seed5.pt) +| 6 | [de_en_seed6.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/direct_models/seed6.pt) + +### Channel Models + +We train on WMT’19 training data. Following [Ng et al., 2019](http://statmt.org/wmt19/pdf/53/WMT33.pdf), we apply language identification filtering and remove sentences longer than 250 tokens as well as sentence pairs with a source/target length ratio exceeding 1.5. This results in 26.8M sentence pairs. + +| Model Size | Seed 4 | Seed 5 | Seed 6 | +|----|----|----|----| +| `big` | [big.seed4.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/big.seed4.pt) | [big.seed5.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/big.seed5.pt) | [big.seed6.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/big.seed6.pt) | +| `big_1_1` | [big_1_1.seed4.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/big_1_1.seed4.pt) | [big_1_1.seed5.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/big_1_1.seed5.pt) | [big_1_1.seed6.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/big_1_1.seed6.pt) | +| `base` | [base.seed4.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/base.seed4.pt) | [base.seed5.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/base.seed5.pt) | [base.seed6.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/base.seed6.pt) | +| `base_1_1` | [base_1_1.seed4.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/base_1_1.seed4.pt) | [base_1_1.seed5.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/base_1_1.seed5.pt) | [base_1_1.seed6.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/base_1_1.seed6.pt) | +| `half` | [half.seed4.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/half.seed4.pt) | [half.seed5.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/half.seed5.pt) | [half.seed6.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/half.seed6.pt) | +| `half_1_1` | [half_1_1.seed4.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/half_1_1.seed4.pt) | [half_1_1.seed5.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/half_1_1.seed5.pt) | [half_1_1.seed6.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/half_1_1.seed6.pt) | +| `quarter` | [quarter.seed4.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/quarter.seed4.pt) | [quarter.seed5.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/quarter.seed5.pt) | [quarter.seed6.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/quarter.seed6.pt) | +| `quarter_1_1` | [quarter_1_1.seed4.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/quarter_1_1.seed4.pt) | [quarter_1_1.seed5.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/quarter_1_1.seed5.pt) | [quarter_1_1.seed6.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/quarter_1_1.seed6.pt) | +| `8th` | [8th.seed4.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/8th.seed4.pt) | [8th.seed5.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/8th.seed5.pt) | [8th.seed6.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/8th.seed6.pt) | +| `8th_1_1` | [8th_1_1.seed4.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/8th_1_1.seed4.pt) | [8th_1_1.seed5.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/8th_1_1.seed5.pt) | [8th_1_1.seed6.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/8th_1_1.seed6.pt) | +| `16th` | [16th.seed4.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/16th.seed4.pt) | [16th.seed5.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/16th.seed5.pt) | [16th.seed6.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/16th.seed6.pt) | +| `16th_1_1` | [16th_1_1.seed4.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/16th_1_1.seed4.pt) | [16th_1_1.seed5.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/16th_1_1.seed5.pt) | [16th_1_1.seed6.pt](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/channel_models/16th_1_1.seed6.pt) | + +### Language Model +The model is trained on de-duplicated English Newscrawl data from 2007-2018 comprising 186 million sentences or 4.5B words after normalization and tokenization. +| | Path | +|----|----| +| `--lm-model` | [transformer_en_lm](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/lm_model/transformer_lm.pt) | +| `--lm-data` | [lm_data](https://dl.fbaipublicfiles.com/fast_noisy_channel/de_en/lm_model/lm_dict/) + + +## Citation + +```bibtex +@inproceedings{bhosale2020language, + title={Language Models not just for Pre-training: Fast Online Neural Noisy Channel Modeling}, + author={Shruti Bhosale and Kyra Yee and Sergey Edunov and Michael Auli}, + booktitle={Proceedings of the Fifth Conference on Machine Translation (WMT)}, + year={2020}, +} + +@inproceedings{yee2019simple, + title={Simple and Effective Noisy Channel Modeling for Neural Machine Translation}, + author={Yee, Kyra and Dauphin, Yann and Auli, Michael}, + booktitle={Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)}, + pages={5700--5705}, + year={2019} +} +``` diff --git a/data/fairseq/examples/fast_noisy_channel/__init__.py b/data/fairseq/examples/fast_noisy_channel/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..9b248c3a24e12ad3da885a7f328c714942de2e6b --- /dev/null +++ b/data/fairseq/examples/fast_noisy_channel/__init__.py @@ -0,0 +1,8 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +from . import noisy_channel_translation # noqa +from . import noisy_channel_sequence_generator # noqa +from . import noisy_channel_beam_search # noqa diff --git a/data/fairseq/examples/fast_noisy_channel/noisy_channel_beam_search.py b/data/fairseq/examples/fast_noisy_channel/noisy_channel_beam_search.py new file mode 100644 index 0000000000000000000000000000000000000000..23869ebcd0c438f36e310c8ccddd3b5c07a71182 --- /dev/null +++ b/data/fairseq/examples/fast_noisy_channel/noisy_channel_beam_search.py @@ -0,0 +1,71 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import torch +from fairseq.search import Search + + +class NoisyChannelBeamSearch(Search): + + def __init__(self, tgt_dict): + super().__init__(tgt_dict) + self.fw_scores_buf = None + self.lm_scores_buf = None + + def _init_buffers(self, t): + # super()._init_buffers(t) + if self.fw_scores_buf is None: + self.scores_buf = t.new() + self.indices_buf = torch.LongTensor().to(device=t.device) + self.beams_buf = torch.LongTensor().to(device=t.device) + self.fw_scores_buf = t.new() + self.lm_scores_buf = t.new() + + def combine_fw_bw(self, combine_method, fw_cum, bw, step): + if combine_method == "noisy_channel": + fw_norm = fw_cum.div(step + 1) + lprobs = bw + fw_norm + elif combine_method == "lm_only": + lprobs = bw + fw_cum + + return lprobs + + def step(self, step, fw_lprobs, scores, bw_lprobs, lm_lprobs, combine_method): + self._init_buffers(fw_lprobs) + bsz, beam_size, vocab_size = fw_lprobs.size() + + if step == 0: + # at the first step all hypotheses are equally likely, so use + # only the first beam + fw_lprobs = fw_lprobs[:, ::beam_size, :].contiguous() + bw_lprobs = bw_lprobs[:, ::beam_size, :].contiguous() + # nothing to add since we are at the first step + fw_lprobs_cum = fw_lprobs + + else: + # make probs contain cumulative scores for each hypothesis + raw_scores = (scores[:, :, step - 1].unsqueeze(-1)) + fw_lprobs_cum = (fw_lprobs.add(raw_scores)) + + combined_lprobs = self.combine_fw_bw(combine_method, fw_lprobs_cum, bw_lprobs, step) + + # choose the top k according to the combined noisy channel model score + torch.topk( + combined_lprobs.view(bsz, -1), + k=min( + # Take the best 2 x beam_size predictions. We'll choose the first + # beam_size of these which don't predict eos to continue with. + beam_size * 2, + combined_lprobs.view(bsz, -1).size(1) - 1, # -1 so we never select pad + ), + out=(self.scores_buf, self.indices_buf), + ) + # save corresponding fw and lm scores + self.fw_scores_buf = torch.gather(fw_lprobs_cum.view(bsz, -1), 1, self.indices_buf) + self.lm_scores_buf = torch.gather(lm_lprobs.view(bsz, -1), 1, self.indices_buf) + # Project back into relative indices and beams + self.beams_buf = self.indices_buf // vocab_size + self.indices_buf.fmod_(vocab_size) + return self.scores_buf, self.fw_scores_buf, self.lm_scores_buf, self.indices_buf, self.beams_buf diff --git a/data/fairseq/examples/fast_noisy_channel/noisy_channel_sequence_generator.py b/data/fairseq/examples/fast_noisy_channel/noisy_channel_sequence_generator.py new file mode 100644 index 0000000000000000000000000000000000000000..ea8fae98e87e9f3e69bc51987703a6429eb0c92a --- /dev/null +++ b/data/fairseq/examples/fast_noisy_channel/noisy_channel_sequence_generator.py @@ -0,0 +1,842 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +from typing import Dict, List, Optional + +import math +import numpy as np + +import torch +import torch.nn.functional as F +from torch import Tensor + +from .noisy_channel_beam_search import NoisyChannelBeamSearch +from fairseq.sequence_generator import EnsembleModel + + +class NoisyChannelSequenceGenerator(object): + def __init__( + self, + combine_method, + tgt_dict, + src_dict=None, + beam_size=1, + max_len_a=0, + max_len_b=200, + min_len=1, + len_penalty=1.0, + unk_penalty=0.0, + retain_dropout=False, + temperature=1.0, + match_source_len=False, + no_repeat_ngram_size=0, + normalize_scores=True, + channel_models=None, + k2=10, + ch_weight=1.0, + channel_scoring_type='log_norm', + top_k_vocab=0, + lm_models=None, + lm_dict=None, + lm_weight=1.0, + normalize_lm_scores_by_tgt_len=False, + ): + """Generates translations of a given source sentence, + using beam search with noisy channel decoding. + + Args: + combine_method (string, optional): Method to combine direct, LM and + channel model scores (default: None) + tgt_dict (~fairseq.data.Dictionary): target dictionary + src_dict (~fairseq.data.Dictionary): source dictionary + beam_size (int, optional): beam width (default: 1) + max_len_a/b (int, optional): generate sequences of maximum length + ax + b, where x is the source length + min_len (int, optional): the minimum length of the generated output + (not including end-of-sentence) + len_penalty (float, optional): length penalty, where <1.0 favors + shorter, >1.0 favors longer sentences (default: 1.0) + unk_penalty (float, optional): unknown word penalty, where <0 + produces more unks, >0 produces fewer (default: 0.0) + retain_dropout (bool, optional): use dropout when generating + (default: False) + temperature (float, optional): temperature, where values + >1.0 produce more uniform samples and values <1.0 produce + sharper samples (default: 1.0) + match_source_len (bool, optional): outputs should match the source + length (default: False) + no_repeat_ngram_size (int, optional): Size of n-grams that we avoid + repeating in the generation (default: 0) + normalize_scores (bool, optional): normalize scores by the length + of the output (default: True) + channel_models (List[~fairseq.models.FairseqModel]): ensemble of models + translating from the target to the source + k2 (int, optional): Top K2 candidates to score per beam at each step (default:10) + ch_weight (int, optional): Weight associated with the channel model score + assuming that the direct model score has weight 1.0 (default: 1.0) + channel_scoring_type (str, optional): String specifying how to score + the channel model (default: 'log_norm') + top_k_vocab (int, optional): If `channel_scoring_type` is `'src_vocab'` or + `'src_vocab_batched'`, then this parameter specifies the number of + most frequent tokens to include in the channel model output vocabulary, + in addition to the source tokens in the input batch (default: 0) + lm_models (List[~fairseq.models.FairseqModel]): ensemble of models + generating text in the target language + lm_dict (~fairseq.data.Dictionary): LM Model dictionary + lm_weight (int, optional): Weight associated with the LM model score + assuming that the direct model score has weight 1.0 (default: 1.0) + normalize_lm_scores_by_tgt_len (bool, optional): Should we normalize LM scores + by the target length? By default, we normalize the combination of + LM and channel model scores by the source length + """ + self.pad = tgt_dict.pad() + self.unk = tgt_dict.unk() + self.eos = tgt_dict.eos() + self.vocab_size = len(tgt_dict) + self.beam_size = beam_size + # the max beam size is the dictionary size - 1, since we never select pad + self.beam_size = min(beam_size, self.vocab_size - 1) + self.max_len_a = max_len_a + self.max_len_b = max_len_b + self.min_len = min_len + self.normalize_scores = normalize_scores + self.len_penalty = len_penalty + self.unk_penalty = unk_penalty + self.retain_dropout = retain_dropout + self.temperature = temperature + self.match_source_len = match_source_len + self.no_repeat_ngram_size = no_repeat_ngram_size + self.channel_models = channel_models + self.src_dict = src_dict + self.tgt_dict = tgt_dict + self.combine_method = combine_method + self.k2 = k2 + self.ch_weight = ch_weight + self.channel_scoring_type = channel_scoring_type + self.top_k_vocab = top_k_vocab + self.lm_models = lm_models + self.lm_dict = lm_dict + self.lm_weight = lm_weight + self.log_softmax_fn = torch.nn.LogSoftmax(dim=1) + self.normalize_lm_scores_by_tgt_len = normalize_lm_scores_by_tgt_len + + self.share_tgt_dict = (self.lm_dict == self.tgt_dict) + self.tgt_to_lm = make_dict2dict(tgt_dict, lm_dict) + + self.ch_scoring_bsz = 3072 + + assert temperature > 0, '--temperature must be greater than 0' + + self.search = NoisyChannelBeamSearch(tgt_dict) + + @torch.no_grad() + def generate( + self, + models, + sample, + prefix_tokens=None, + bos_token=None, + **kwargs + ): + """Generate a batch of translations. + Args: + models (List[~fairseq.models.FairseqModel]): ensemble of models + sample (dict): batch + prefix_tokens (torch.LongTensor, optional): force decoder to begin + with these tokens + """ + model = EnsembleModel(models) + incremental_states = torch.jit.annotate( + List[Dict[str, Dict[str, Optional[Tensor]]]], + [ + torch.jit.annotate(Dict[str, Dict[str, Optional[Tensor]]], {}) + for i in range(model.models_size) + ], + ) + if not self.retain_dropout: + model.eval() + + # model.forward normally channels prev_output_tokens into the decoder + # separately, but SequenceGenerator directly calls model.encoder + encoder_input = { + k: v for k, v in sample['net_input'].items() + if k != 'prev_output_tokens' + } + src_tokens = encoder_input['src_tokens'] + src_lengths_no_eos = (src_tokens.ne(self.eos) & src_tokens.ne(self.pad)).long().sum(dim=1) + input_size = src_tokens.size() + # batch dimension goes first followed by source lengths + bsz = input_size[0] + src_len = input_size[1] + beam_size = self.beam_size + + if self.match_source_len: + max_len = src_lengths_no_eos.max().item() + else: + max_len = min( + int(self.max_len_a * src_len + self.max_len_b), + # exclude the EOS marker + model.max_decoder_positions() - 1, + ) + + # compute the encoder output for each beam + encoder_outs = model.forward_encoder(encoder_input) + new_order = torch.arange(bsz).view(-1, 1).repeat(1, beam_size).view(-1) + new_order = new_order.to(src_tokens.device).long() + encoder_outs = model.reorder_encoder_out(encoder_outs, new_order) + + src_lengths = encoder_input['src_lengths'] + # initialize buffers + scores = src_tokens.new(bsz * beam_size, max_len + 1).float().fill_(0) + lm_prefix_scores = src_tokens.new(bsz * beam_size).float().fill_(0) + + scores_buf = scores.clone() + tokens = src_tokens.new(bsz * beam_size, max_len + 2).long().fill_(self.pad) + tokens_buf = tokens.clone() + tokens[:, 0] = self.eos if bos_token is None else bos_token + + # reorder source tokens so they may be used as a reference in generating P(S|T) + src_tokens = reorder_all_tokens(src_tokens, src_lengths, self.src_dict.eos_index) + + src_tokens = src_tokens.repeat(1, beam_size).view(-1, src_len) + src_lengths = src_lengths.view(bsz, -1).repeat(1, beam_size).view(bsz*beam_size, -1) + + attn, attn_buf = None, None + nonpad_idxs = None + + # The cands_to_ignore indicates candidates that should be ignored. + # For example, suppose we're sampling and have already finalized 2/5 + # samples. Then the cands_to_ignore would mark 2 positions as being ignored, + # so that we only finalize the remaining 3 samples. + cands_to_ignore = src_tokens.new_zeros(bsz, beam_size).eq(-1) # forward and backward-compatible False mask + + # list of completed sentences + finalized = [[] for i in range(bsz)] + finished = [False for i in range(bsz)] + num_remaining_sent = bsz + + # number of candidate hypos per step + cand_size = 2 * beam_size # 2 x beam size in case half are EOS + + # offset arrays for converting between different indexing schemes + bbsz_offsets = (torch.arange(0, bsz) * beam_size).unsqueeze(1).type_as(tokens) + cand_offsets = torch.arange(0, cand_size).type_as(tokens) + + # helper function for allocating buffers on the fly + buffers = {} + + def buffer(name, type_of=tokens): # noqa + if name not in buffers: + buffers[name] = type_of.new() + return buffers[name] + + def is_finished(sent, step, unfin_idx): + """ + Check whether we've finished generation for a given sentence, by + comparing the worst score among finalized hypotheses to the best + possible score among unfinalized hypotheses. + """ + assert len(finalized[sent]) <= beam_size + if len(finalized[sent]) == beam_size: + return True + return False + + def finalize_hypos(step, bbsz_idx, eos_scores, combined_noisy_channel_eos_scores): + """ + Finalize the given hypotheses at this step, while keeping the total + number of finalized hypotheses per sentence <= beam_size. + + Note: the input must be in the desired finalization order, so that + hypotheses that appear earlier in the input are preferred to those + that appear later. + + Args: + step: current time step + bbsz_idx: A vector of indices in the range [0, bsz*beam_size), + indicating which hypotheses to finalize + eos_scores: A vector of the same size as bbsz_idx containing + fw scores for each hypothesis + combined_noisy_channel_eos_scores: A vector of the same size as bbsz_idx containing + combined noisy channel scores for each hypothesis + """ + assert bbsz_idx.numel() == eos_scores.numel() + + # clone relevant token and attention tensors + tokens_clone = tokens.index_select(0, bbsz_idx) + tokens_clone = tokens_clone[:, 1:step + 2] # skip the first index, which is EOS + assert not tokens_clone.eq(self.eos).any() + tokens_clone[:, step] = self.eos + attn_clone = attn.index_select(0, bbsz_idx)[:, :, 1:step+2] if attn is not None else None + + # compute scores per token position + pos_scores = scores.index_select(0, bbsz_idx)[:, :step+1] + pos_scores[:, step] = eos_scores + # convert from cumulative to per-position scores + pos_scores[:, 1:] = pos_scores[:, 1:] - pos_scores[:, :-1] + + # normalize sentence-level scores + if self.normalize_scores: + combined_noisy_channel_eos_scores /= (step + 1) ** self.len_penalty + + cum_unfin = [] + prev = 0 + for f in finished: + if f: + prev += 1 + else: + cum_unfin.append(prev) + + sents_seen = set() + for i, (idx, score) in enumerate(zip(bbsz_idx.tolist(), combined_noisy_channel_eos_scores.tolist())): + unfin_idx = idx // beam_size + sent = unfin_idx + cum_unfin[unfin_idx] + + sents_seen.add((sent, unfin_idx)) + + if self.match_source_len and step > src_lengths_no_eos[unfin_idx]: + score = -math.inf + + def get_hypo(): + + if attn_clone is not None: + # remove padding tokens from attn scores + hypo_attn = attn_clone[i][nonpad_idxs[sent]] + _, alignment = hypo_attn.max(dim=0) + else: + hypo_attn = None + alignment = None + + return { + 'tokens': tokens_clone[i], + 'score': score, + 'attention': hypo_attn, # src_len x tgt_len + 'alignment': alignment, + 'positional_scores': pos_scores[i], + } + + if len(finalized[sent]) < beam_size: + finalized[sent].append(get_hypo()) + + newly_finished = [] + for sent, unfin_idx in sents_seen: + # check termination conditions for this sentence + if not finished[sent] and is_finished(sent, step, unfin_idx): + finished[sent] = True + newly_finished.append(unfin_idx) + return newly_finished + + def noisy_channel_rescoring(lprobs, beam_size, bsz, src_tokens, tokens, k): + """Rescore the top k hypothesis from each beam using noisy channel modeling + Returns: + new_fw_lprobs: the direct model probabilities after pruning the top k + new_ch_lm_lprobs: the combined channel and language model probabilities + new_lm_lprobs: the language model probabilities after pruning the top k + """ + with torch.no_grad(): + lprobs_size = lprobs.size() + if prefix_tokens is not None and step < prefix_tokens.size(1): + probs_slice = lprobs.view(bsz, -1, lprobs.size(-1))[:, 0, :] + cand_scores = torch.gather( + probs_slice, dim=1, + index=prefix_tokens[:, step].view(-1, 1).data + ).expand(-1, beam_size).contiguous().view(bsz*beam_size, 1) + cand_indices = prefix_tokens[:, step].view(-1, 1).expand(bsz, beam_size).data.contiguous().view(bsz*beam_size, 1) + + # need to calculate and save fw and lm probs for prefix tokens + fw_top_k = cand_scores + fw_top_k_idx = cand_indices + k = 1 + else: + # take the top k best words for every sentence in batch*beam + fw_top_k, fw_top_k_idx = torch.topk(lprobs.view(beam_size*bsz, -1), k=k) + eos_idx = torch.nonzero(fw_top_k_idx.view(bsz*beam_size*k, -1) == self.eos)[:, 0] + ch_scores = fw_top_k.new_full((beam_size*bsz*k, ), 0) + src_size = torch.sum(src_tokens[:, :] != self.src_dict.pad_index, dim=1, keepdim=True, dtype=fw_top_k.dtype) + + if self.combine_method != "lm_only": + temp_src_tokens_full = src_tokens[:, :].repeat(1, k).view(bsz*beam_size*k, -1) + not_padding = temp_src_tokens_full[:, 1:] != self.src_dict.pad_index + cur_tgt_size = step+2 + + # add eos to all candidate sentences except those that already end in eos + eos_tokens = tokens[:, 0].repeat(1, k).view(-1, 1) + eos_tokens[eos_idx] = self.tgt_dict.pad_index + + if step == 0: + channel_input = torch.cat((fw_top_k_idx.view(-1, 1), eos_tokens), 1) + else: + # move eos from beginning to end of target sentence + channel_input = torch.cat((tokens[:, 1:step + 1].repeat(1, k).view(-1, step), fw_top_k_idx.view(-1, 1), eos_tokens), 1) + + ch_input_lengths = torch.tensor(np.full(channel_input.size(0), cur_tgt_size)) + ch_input_lengths[eos_idx] = cur_tgt_size-1 + if self.channel_scoring_type == "unnormalized": + ch_encoder_output = channel_model.encoder(channel_input, src_lengths=ch_input_lengths) + ch_decoder_output, _ = channel_model.decoder(temp_src_tokens_full, encoder_out=ch_encoder_output, features_only=True) + del ch_encoder_output + ch_intermed_scores = channel_model.decoder.unnormalized_scores_given_target(ch_decoder_output, target_ids=temp_src_tokens_full[:, 1:]) + ch_intermed_scores = ch_intermed_scores.float() + ch_intermed_scores *= not_padding.float() + ch_scores = torch.sum(ch_intermed_scores, dim=1) + elif self.channel_scoring_type == "k2_separate": + for k_idx in range(k): + k_eos_tokens = eos_tokens[k_idx::k, :] + if step == 0: + k_ch_input = torch.cat((fw_top_k_idx[:, k_idx:k_idx+1], k_eos_tokens), 1) + else: + # move eos from beginning to end of target sentence + k_ch_input = torch.cat((tokens[:, 1:step + 1], fw_top_k_idx[:, k_idx:k_idx+1], k_eos_tokens), 1) + k_ch_input_lengths = ch_input_lengths[k_idx::k] + k_ch_output = channel_model(k_ch_input, k_ch_input_lengths, src_tokens) + k_ch_lprobs = channel_model.get_normalized_probs(k_ch_output, log_probs=True) + k_ch_intermed_scores = torch.gather(k_ch_lprobs[:, :-1, :], 2, src_tokens[:, 1:].unsqueeze(2)).squeeze(2) + k_ch_intermed_scores *= not_padding.float() + ch_scores[k_idx::k] = torch.sum(k_ch_intermed_scores, dim=1) + elif self.channel_scoring_type == "src_vocab": + ch_encoder_output = channel_model.encoder(channel_input, src_lengths=ch_input_lengths) + ch_decoder_output, _ = channel_model.decoder(temp_src_tokens_full, encoder_out=ch_encoder_output, features_only=True) + + del ch_encoder_output + ch_lprobs = normalized_scores_with_batch_vocab( + channel_model.decoder, + ch_decoder_output, src_tokens, k, bsz, beam_size, + self.src_dict.pad_index, top_k=self.top_k_vocab) + ch_scores = torch.sum(ch_lprobs, dim=1) + elif self.channel_scoring_type == "src_vocab_batched": + ch_bsz_size = temp_src_tokens_full.shape[0] + ch_lprobs_list = [None] * len(range(0, ch_bsz_size, self.ch_scoring_bsz)) + for i, start_idx in enumerate(range(0, ch_bsz_size, self.ch_scoring_bsz)): + end_idx = min(start_idx + self.ch_scoring_bsz, ch_bsz_size) + temp_src_tokens_full_batch = temp_src_tokens_full[start_idx:end_idx, :] + channel_input_batch = channel_input[start_idx:end_idx, :] + ch_input_lengths_batch = ch_input_lengths[start_idx:end_idx] + ch_encoder_output_batch = channel_model.encoder(channel_input_batch, src_lengths=ch_input_lengths_batch) + ch_decoder_output_batch, _ = channel_model.decoder(temp_src_tokens_full_batch, encoder_out=ch_encoder_output_batch, features_only=True) + ch_lprobs_list[i] = normalized_scores_with_batch_vocab( + channel_model.decoder, + ch_decoder_output_batch, src_tokens, k, bsz, beam_size, + self.src_dict.pad_index, top_k=self.top_k_vocab, + start_idx=start_idx, end_idx=end_idx) + ch_lprobs = torch.cat(ch_lprobs_list, dim=0) + ch_scores = torch.sum(ch_lprobs, dim=1) + else: + ch_output = channel_model(channel_input, ch_input_lengths, temp_src_tokens_full) + ch_lprobs = channel_model.get_normalized_probs(ch_output, log_probs=True) + ch_intermed_scores = torch.gather(ch_lprobs[:, :-1, :], 2, temp_src_tokens_full[:, 1:].unsqueeze(2)).squeeze().view(bsz*beam_size*k, -1) + ch_intermed_scores *= not_padding.float() + ch_scores = torch.sum(ch_intermed_scores, dim=1) + + else: + cur_tgt_size = 0 + ch_scores = ch_scores.view(bsz*beam_size, k) + expanded_lm_prefix_scores = lm_prefix_scores.unsqueeze(1).expand(-1, k).flatten() + + if self.share_tgt_dict: + lm_scores = get_lm_scores(lm, tokens[:, :step + 1].view(-1, step+1), lm_incremental_states, fw_top_k_idx.view(-1, 1), torch.tensor(np.full(tokens.size(0), step+1)), k) + else: + new_lm_input = dict2dict(tokens[:, :step + 1].view(-1, step+1), self.tgt_to_lm) + new_cands = dict2dict(fw_top_k_idx.view(-1, 1), self.tgt_to_lm) + lm_scores = get_lm_scores(lm, new_lm_input, lm_incremental_states, new_cands, torch.tensor(np.full(tokens.size(0), step+1)), k) + + lm_scores.add_(expanded_lm_prefix_scores) + ch_lm_scores = combine_ch_lm(self.combine_method, ch_scores, lm_scores, src_size, cur_tgt_size) + # initialize all as min value + new_fw_lprobs = ch_scores.new(lprobs_size).fill_(-1e17).view(bsz*beam_size, -1) + new_ch_lm_lprobs = ch_scores.new(lprobs_size).fill_(-1e17).view(bsz*beam_size, -1) + new_lm_lprobs = ch_scores.new(lprobs_size).fill_(-1e17).view(bsz*beam_size, -1) + new_fw_lprobs[:, self.pad] = -math.inf + new_ch_lm_lprobs[:, self.pad] = -math.inf + new_lm_lprobs[:, self.pad] = -math.inf + + new_fw_lprobs.scatter_(1, fw_top_k_idx, fw_top_k) + new_ch_lm_lprobs.scatter_(1, fw_top_k_idx, ch_lm_scores) + new_lm_lprobs.scatter_(1, fw_top_k_idx, lm_scores.view(-1, k)) + return new_fw_lprobs, new_ch_lm_lprobs, new_lm_lprobs + + def combine_ch_lm(combine_type, ch_scores, lm_scores1, src_size, tgt_size): + if self.channel_scoring_type == "unnormalized": + ch_scores = self.log_softmax_fn( + ch_scores.view(-1, self.beam_size * self.k2) + ).view(ch_scores.shape) + ch_scores = ch_scores * self.ch_weight + lm_scores1 = lm_scores1 * self.lm_weight + + if combine_type == "lm_only": + # log P(T|S) + log P(T) + ch_scores = lm_scores1.view(ch_scores.size()) + elif combine_type == "noisy_channel": + # 1/t log P(T|S) + 1/s log P(S|T) + 1/t log P(T) + if self.normalize_lm_scores_by_tgt_len: + ch_scores.div_(src_size) + lm_scores_norm = lm_scores1.view(ch_scores.size()).div(tgt_size) + ch_scores.add_(lm_scores_norm) + # 1/t log P(T|S) + 1/s log P(S|T) + 1/s log P(T) + else: + ch_scores.add_(lm_scores1.view(ch_scores.size())) + ch_scores.div_(src_size) + + return ch_scores + + if self.channel_models is not None: + channel_model = self.channel_models[0] # assume only one channel_model model + else: + channel_model = None + + lm = EnsembleModel(self.lm_models) + lm_incremental_states = torch.jit.annotate( + List[Dict[str, Dict[str, Optional[Tensor]]]], + [ + torch.jit.annotate(Dict[str, Dict[str, Optional[Tensor]]], {}) + for i in range(lm.models_size) + ], + ) + + reorder_state = None + batch_idxs = None + for step in range(max_len + 1): # one extra step for EOS marker + # reorder decoder internal states based on the prev choice of beams + if reorder_state is not None: + if batch_idxs is not None: + # update beam indices to take into account removed sentences + corr = batch_idxs - torch.arange(batch_idxs.numel()).type_as(batch_idxs) + reorder_state.view(-1, beam_size).add_(corr.unsqueeze(-1) * beam_size) + model.reorder_incremental_state(incremental_states, reorder_state) + encoder_outs = model.reorder_encoder_out(encoder_outs, reorder_state) + + lm.reorder_incremental_state(lm_incremental_states, reorder_state) + + fw_lprobs, avg_attn_scores = model.forward_decoder( + tokens[:, :step + 1], encoder_outs, incremental_states, temperature=self.temperature, + ) + + fw_lprobs[:, self.pad] = -math.inf # never select pad + fw_lprobs[:, self.unk] -= self.unk_penalty # apply unk penalty + fw_lprobs, ch_lm_lprobs, lm_lprobs = noisy_channel_rescoring(fw_lprobs, beam_size, bsz, src_tokens, tokens, self.k2) + + # handle min and max length constraints + if step >= max_len: + fw_lprobs[:, :self.eos] = -math.inf + fw_lprobs[:, self.eos + 1:] = -math.inf + elif step < self.min_len: + fw_lprobs[:, self.eos] = -math.inf + + # handle prefix tokens (possibly with different lengths) + if prefix_tokens is not None and step < prefix_tokens.size(1): + prefix_toks = prefix_tokens[:, step].unsqueeze(-1).repeat(1, beam_size).view(-1) + prefix_mask = prefix_toks.ne(self.pad) + + prefix_fw_lprobs = fw_lprobs.gather(-1, prefix_toks.unsqueeze(-1)) + fw_lprobs[prefix_mask] = -math.inf + fw_lprobs[prefix_mask] = fw_lprobs[prefix_mask].scatter_( + -1, prefix_toks[prefix_mask].unsqueeze(-1), prefix_fw_lprobs + ) + + prefix_ch_lm_lprobs = ch_lm_lprobs.gather(-1, prefix_toks.unsqueeze(-1)) + ch_lm_lprobs[prefix_mask] = -math.inf + ch_lm_lprobs[prefix_mask] = ch_lm_lprobs[prefix_mask].scatter_( + -1, prefix_toks[prefix_mask].unsqueeze(-1), prefix_ch_lm_lprobs + ) + + prefix_lm_lprobs = lm_lprobs.gather(-1, prefix_toks.unsqueeze(-1)) + lm_lprobs[prefix_mask] = -math.inf + lm_lprobs[prefix_mask] = lm_lprobs[prefix_mask].scatter_( + -1, prefix_toks[prefix_mask].unsqueeze(-1), prefix_lm_lprobs + ) + + # if prefix includes eos, then we should make sure tokens and + # scores are the same across all beams + eos_mask = prefix_toks.eq(self.eos) + if eos_mask.any(): + # validate that the first beam matches the prefix + first_beam = tokens[eos_mask].view(-1, beam_size, tokens.size(-1))[:, 0, 1:step + 1] + eos_mask_batch_dim = eos_mask.view(-1, beam_size)[:, 0] + target_prefix = prefix_tokens[eos_mask_batch_dim][:, :step] + assert (first_beam == target_prefix).all() + + def replicate_first_beam(tensor, mask): + tensor = tensor.view(-1, beam_size, tensor.size(-1)) + tensor[mask] = tensor[mask][:, :1, :] + return tensor.view(-1, tensor.size(-1)) + + # copy tokens, scores and lprobs from the first beam to all beams + tokens = replicate_first_beam(tokens, eos_mask_batch_dim) + scores = replicate_first_beam(scores, eos_mask_batch_dim) + + fw_lprobs = replicate_first_beam(fw_lprobs, eos_mask_batch_dim) + ch_lm_lprobs = replicate_first_beam(ch_lm_lprobs, eos_mask_batch_dim) + lm_lprobs = replicate_first_beam(lm_lprobs, eos_mask_batch_dim) + + if self.no_repeat_ngram_size > 0: + # for each beam and batch sentence, generate a list of previous ngrams + gen_ngrams = [{} for bbsz_idx in range(bsz * beam_size)] + for bbsz_idx in range(bsz * beam_size): + gen_tokens = tokens[bbsz_idx].tolist() + for ngram in zip(*[gen_tokens[i:] for i in range(self.no_repeat_ngram_size)]): + gen_ngrams[bbsz_idx][tuple(ngram[:-1])] = \ + gen_ngrams[bbsz_idx].get(tuple(ngram[:-1]), []) + [ngram[-1]] + + # Record attention scores + if avg_attn_scores is not None: + if attn is None: + attn = scores.new(bsz * beam_size, src_tokens.size(1), max_len + 2) + attn_buf = attn.clone() + nonpad_idxs = src_tokens.ne(self.pad) + attn[:, :, step + 1].copy_(avg_attn_scores) + + scores = scores.type_as(fw_lprobs) + scores_buf = scores_buf.type_as(fw_lprobs) + + self.search.set_src_lengths(src_lengths_no_eos) + + if self.no_repeat_ngram_size > 0: + def calculate_banned_tokens(bbsz_idx): + # before decoding the next token, prevent decoding of ngrams that have already appeared + ngram_index = tuple(tokens[bbsz_idx, step + 2 - self.no_repeat_ngram_size:step + 1].tolist()) + return gen_ngrams[bbsz_idx].get(ngram_index, []) + + if step + 2 - self.no_repeat_ngram_size >= 0: + # no banned tokens if we haven't generated no_repeat_ngram_size tokens yet + banned_tokens = [calculate_banned_tokens(bbsz_idx) for bbsz_idx in range(bsz * beam_size)] + else: + banned_tokens = [[] for bbsz_idx in range(bsz * beam_size)] + + for bbsz_idx in range(bsz * beam_size): + fw_lprobs[bbsz_idx, banned_tokens[bbsz_idx]] = -math.inf + + combined_noisy_channel_scores, fw_lprobs_top_k, lm_lprobs_top_k, cand_indices, cand_beams = self.search.step( + step, + fw_lprobs.view(bsz, -1, self.vocab_size), + scores.view(bsz, beam_size, -1)[:, :, :step], ch_lm_lprobs.view(bsz, -1, self.vocab_size), + lm_lprobs.view(bsz, -1, self.vocab_size), self.combine_method + ) + + # cand_bbsz_idx contains beam indices for the top candidate + # hypotheses, with a range of values: [0, bsz*beam_size), + # and dimensions: [bsz, cand_size] + cand_bbsz_idx = cand_beams.add(bbsz_offsets) + + # finalize hypotheses that end in eos (except for candidates to be ignored) + eos_mask = cand_indices.eq(self.eos) + eos_mask[:, :beam_size] &= ~cands_to_ignore + + # only consider eos when it's among the top beam_size indices + eos_bbsz_idx = torch.masked_select( + cand_bbsz_idx[:, :beam_size], mask=eos_mask[:, :beam_size] + ) + + finalized_sents = set() + if eos_bbsz_idx.numel() > 0: + eos_scores = torch.masked_select( + fw_lprobs_top_k[:, :beam_size], mask=eos_mask[:, :beam_size] + ) + combined_noisy_channel_eos_scores = torch.masked_select( + combined_noisy_channel_scores[:, :beam_size], + mask=eos_mask[:, :beam_size], + ) + + # finalize hypo using channel model score + finalized_sents = finalize_hypos( + step, eos_bbsz_idx, eos_scores, combined_noisy_channel_eos_scores) + + num_remaining_sent -= len(finalized_sents) + + assert num_remaining_sent >= 0 + if num_remaining_sent == 0: + break + + if len(finalized_sents) > 0: + new_bsz = bsz - len(finalized_sents) + + # construct batch_idxs which holds indices of batches to keep for the next pass + batch_mask = cand_indices.new_ones(bsz) + batch_mask[cand_indices.new(finalized_sents)] = 0 + batch_idxs = torch.nonzero(batch_mask).squeeze(-1) + + eos_mask = eos_mask[batch_idxs] + cand_beams = cand_beams[batch_idxs] + bbsz_offsets.resize_(new_bsz, 1) + cand_bbsz_idx = cand_beams.add(bbsz_offsets) + + lm_lprobs_top_k = lm_lprobs_top_k[batch_idxs] + + fw_lprobs_top_k = fw_lprobs_top_k[batch_idxs] + cand_indices = cand_indices[batch_idxs] + if prefix_tokens is not None: + prefix_tokens = prefix_tokens[batch_idxs] + src_lengths_no_eos = src_lengths_no_eos[batch_idxs] + cands_to_ignore = cands_to_ignore[batch_idxs] + + scores = scores.view(bsz, -1)[batch_idxs].view(new_bsz * beam_size, -1) + scores_buf.resize_as_(scores) + tokens = tokens.view(bsz, -1)[batch_idxs].view(new_bsz * beam_size, -1) + tokens_buf.resize_as_(tokens) + src_tokens = src_tokens.view(bsz, -1)[batch_idxs].view(new_bsz * beam_size, -1) + src_lengths = src_lengths.view(bsz, -1)[batch_idxs].view(new_bsz * beam_size, -1) + lm_prefix_scores = lm_prefix_scores.view(bsz, -1)[batch_idxs].view(new_bsz * beam_size, -1).squeeze() + + if attn is not None: + attn = attn.view(bsz, -1)[batch_idxs].view(new_bsz * beam_size, attn.size(1), -1) + attn_buf.resize_as_(attn) + bsz = new_bsz + else: + batch_idxs = None + + # Set active_mask so that values > cand_size indicate eos or + # ignored hypos and values < cand_size indicate candidate + # active hypos. After this, the min values per row are the top + # candidate active hypos. + eos_mask[:, :beam_size] |= cands_to_ignore + active_mask = torch.add( + eos_mask.type_as(cand_offsets) * cand_size, + cand_offsets[: eos_mask.size(1)], + ) + + # get the top beam_size active hypotheses, which are just the hypos + # with the smallest values in active_mask + active_hypos, new_cands_to_ignore = buffer('active_hypos'), buffer('new_cands_to_ignore') + torch.topk( + active_mask, k=beam_size, dim=1, largest=False, + out=(new_cands_to_ignore, active_hypos) + ) + + # update cands_to_ignore to ignore any finalized hypos + cands_to_ignore = new_cands_to_ignore.ge(cand_size)[:, :beam_size] + assert (~cands_to_ignore).any(dim=1).all() + + active_bbsz_idx = buffer('active_bbsz_idx') + torch.gather( + cand_bbsz_idx, dim=1, index=active_hypos, + out=active_bbsz_idx, + ) + active_scores = torch.gather( + fw_lprobs_top_k, dim=1, index=active_hypos, + out=scores[:, step].view(bsz, beam_size), + ) + + active_bbsz_idx = active_bbsz_idx.view(-1) + active_scores = active_scores.view(-1) + + # copy tokens and scores for active hypotheses + torch.index_select( + tokens[:, :step + 1], dim=0, index=active_bbsz_idx, + out=tokens_buf[:, :step + 1], + ) + torch.gather( + cand_indices, dim=1, index=active_hypos, + out=tokens_buf.view(bsz, beam_size, -1)[:, :, step + 1], + ) + if step > 0: + torch.index_select( + scores[:, :step], dim=0, index=active_bbsz_idx, + out=scores_buf[:, :step], + ) + torch.gather( + fw_lprobs_top_k, dim=1, index=active_hypos, + out=scores_buf.view(bsz, beam_size, -1)[:, :, step], + ) + torch.gather( + lm_lprobs_top_k, dim=1, index=active_hypos, + out=lm_prefix_scores.view(bsz, beam_size) + ) + + # copy attention for active hypotheses + if attn is not None: + torch.index_select( + attn[:, :, :step + 2], dim=0, index=active_bbsz_idx, + out=attn_buf[:, :, :step + 2], + ) + + # swap buffers + tokens, tokens_buf = tokens_buf, tokens + scores, scores_buf = scores_buf, scores + if attn is not None: + attn, attn_buf = attn_buf, attn + + # reorder incremental state in decoder + reorder_state = active_bbsz_idx + + # sort by score descending + for sent in range(len(finalized)): + finalized[sent] = sorted(finalized[sent], key=lambda r: r['score'], reverse=True) + + return finalized + + +def get_lm_scores(model, input_tokens, incremental_states, cand_tokens, input_len, k): + with torch.no_grad(): + lm_lprobs, avg_attn_scores = model.forward_decoder( + input_tokens, encoder_outs=None, incremental_states=incremental_states, + ) + + lm_lprobs_size = lm_lprobs.size(0) + probs_next_wrd = torch.gather(lm_lprobs.repeat(1, k).view(lm_lprobs_size*k, -1), 1, cand_tokens).squeeze().view(-1) + + return probs_next_wrd + + +def make_dict2dict(old_dict, new_dict): + dict2dict_map = {} + for sym in old_dict.symbols: + dict2dict_map[old_dict.index(sym)] = new_dict.index(sym) + return dict2dict_map + + +def dict2dict(tokens, dict2dict_map): + if tokens.device == torch.device('cpu'): + tokens_tmp = tokens + else: + tokens_tmp = tokens.cpu() + return tokens_tmp.map_( + tokens_tmp, + lambda _, val, dict2dict_map=dict2dict_map : dict2dict_map[float(val)] + ).to(tokens.device) + + +def reorder_tokens(tokens, lengths, eos): + # reorder source tokens so they may be used as reference for P(S|T) + return torch.cat((tokens.new([eos]), tokens[-lengths:-1], tokens[:-lengths]), 0) + + +def reorder_all_tokens(tokens, lengths, eos): + # used to reorder src tokens from [ .. ] to [ ...] + # so source tokens can be used to predict P(S|T) + return torch.stack([reorder_tokens(token, length, eos) for token, length in zip(tokens, lengths)]) + + +def normalized_scores_with_batch_vocab( + model_decoder, features, target_ids, k, bsz, beam_size, + pad_idx, top_k=0, vocab_size_meter=None, start_idx=None, + end_idx=None, **kwargs): + """ + Get normalized probabilities (or log probs) from a net's output + w.r.t. vocab consisting of target IDs in the batch + """ + if model_decoder.adaptive_softmax is None: + weight = model_decoder.output_projection.weight + vocab_ids = torch.unique( + torch.cat( + (torch.unique(target_ids), torch.arange(top_k, device=target_ids.device)) + ) + ) + id_map = dict(zip(vocab_ids.tolist(), range(len(vocab_ids)))) + mapped_target_ids = target_ids.cpu().apply_( + lambda x, id_map=id_map: id_map[x] + ).to(target_ids.device) + expanded_target_ids = mapped_target_ids[:, :].repeat(1, k).view(bsz*beam_size*k, -1) + if start_idx is not None and end_idx is not None: + expanded_target_ids = expanded_target_ids[start_idx:end_idx, :] + logits = F.linear(features, weight[vocab_ids, :]) + log_softmax = F.log_softmax(logits, dim=-1, dtype=torch.float32) + intermed_scores = torch.gather( + log_softmax[:, :-1, :], + 2, + expanded_target_ids[:, 1:].unsqueeze(2), + ).squeeze() + not_padding = expanded_target_ids[:, 1:] != pad_idx + intermed_scores *= not_padding.float() + return intermed_scores + else: + raise ValueError("adaptive softmax doesn't work with " + + "`normalized_scores_with_batch_vocab()`") diff --git a/data/fairseq/examples/fast_noisy_channel/noisy_channel_translation.py b/data/fairseq/examples/fast_noisy_channel/noisy_channel_translation.py new file mode 100644 index 0000000000000000000000000000000000000000..b74bdfd456f9b7c546ce528173c77431b4f57ac1 --- /dev/null +++ b/data/fairseq/examples/fast_noisy_channel/noisy_channel_translation.py @@ -0,0 +1,127 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +from fairseq.tasks.translation import TranslationTask +from fairseq.tasks.language_modeling import LanguageModelingTask +from fairseq import checkpoint_utils +import argparse +from fairseq.tasks import register_task +import torch + + +@register_task("noisy_channel_translation") +class NoisyChannelTranslation(TranslationTask): + """ + Rescore the top k candidates from each beam using noisy channel modeling + """ + + @staticmethod + def add_args(parser): + """Add task-specific arguments to the parser.""" + TranslationTask.add_args(parser) + # fmt: off + parser.add_argument('--channel-model', metavar='FILE', + help='path to P(S|T) model. P(S|T) and P(T|S) must share source and target dictionaries.') + parser.add_argument('--combine-method', default='lm_only', + choices=['lm_only', 'noisy_channel'], + help="""method for combining direct and channel model scores. + lm_only: decode with P(T|S)P(T) + noisy_channel: decode with 1/t P(T|S) + 1/s(P(S|T)P(T))""") + parser.add_argument('--normalize-lm-scores-by-tgt-len', action='store_true', default=False, + help='normalize lm score by target length instead of source length') + parser.add_argument('--channel-scoring-type', default='log_norm', choices=['unnormalized', 'log_norm', 'k2_separate', 'src_vocab', 'src_vocab_batched'], + help="Normalize bw scores with log softmax or return bw scores without log softmax") + parser.add_argument('--top-k-vocab', default=0, type=int, + help='top k vocab IDs to use with `src_vocab` in channel model scoring') + parser.add_argument('--k2', default=50, type=int, + help='the top k2 candidates to rescore with the noisy channel model for each beam') + parser.add_argument('--ch-wt', default=1, type=float, + help='weight for the channel model') + parser.add_argument('--lm-model', metavar='FILE', + help='path to lm model file, to model P(T). P(T) must share the same vocab as the direct model on the target side') + parser.add_argument('--lm-data', metavar='FILE', + help='path to lm model training data for target language, used to properly load LM with correct dictionary') + parser.add_argument('--lm-wt', default=1, type=float, + help='the weight of the lm in joint decoding') + # fmt: on + + def build_generator( + self, models, args, seq_gen_cls=None, extra_gen_cls_kwargs=None + ): + if getattr(args, "score_reference", False): + raise NotImplementedError() + else: + from .noisy_channel_sequence_generator import NoisyChannelSequenceGenerator + use_cuda = torch.cuda.is_available() and not self.args.cpu + assert self.args.lm_model is not None, '--lm-model required for noisy channel generation!' + assert self.args.lm_data is not None, '--lm-data required for noisy channel generation to map between LM and bitext vocabs' + if self.args.channel_model is not None: + import copy + ch_args_task = copy.deepcopy(self.args) + tmp = ch_args_task.source_lang + ch_args_task.source_lang = ch_args_task.target_lang + ch_args_task.target_lang = tmp + ch_args_task._name = 'translation' + channel_task = TranslationTask.setup_task(ch_args_task) + + arg_dict = {} + arg_dict['task'] = 'language_modeling' + arg_dict['sample_break_mode'] = 'eos' + arg_dict['data'] = self.args.lm_data + arg_dict['output_dictionary_size'] = -1 + lm_args = argparse.Namespace(**arg_dict) + lm_task = LanguageModelingTask.setup_task(lm_args) + lm_dict = lm_task.output_dictionary + + if self.args.channel_model is not None: + channel_models, _ = checkpoint_utils.load_model_ensemble(self.args.channel_model.split(':'), task=channel_task) + + for model in channel_models: + model.make_generation_fast_( + beamable_mm_beam_size=None if args.no_beamable_mm else args.beam, + need_attn=args.print_alignment, + ) + if self.args.fp16: + model.half() + if use_cuda: + model.cuda() + else: + channel_models = None + + lm_models, _ = checkpoint_utils.load_model_ensemble(self.args.lm_model.split(':'), task=lm_task) + + for model in lm_models: + model.make_generation_fast_( + beamable_mm_beam_size=None if args.no_beamable_mm else args.beam, + need_attn=args.print_alignment, + ) + if self.args.fp16: + model.half() + if use_cuda: + model.cuda() + return NoisyChannelSequenceGenerator( + combine_method=self.args.combine_method, + tgt_dict=self.target_dictionary, + src_dict=self.source_dictionary, + beam_size=getattr(args, 'beam', 5), + max_len_a=getattr(args, 'max_len_a', 0), + max_len_b=getattr(args, 'max_len_b', 200), + min_len=getattr(args, 'min_len', 1), + len_penalty=getattr(args, 'lenpen', 1), + unk_penalty=getattr(args, 'unkpen', 0), + temperature=getattr(args, 'temperature', 1.), + match_source_len=getattr(args, 'match_source_len', False), + no_repeat_ngram_size=getattr(args, 'no_repeat_ngram_size', 0), + normalize_scores=(not getattr(args, 'unnormalized', False)), + channel_models=channel_models, + k2=getattr(self.args, 'k2', 50), + ch_weight=getattr(self.args, 'ch_wt', 1), + channel_scoring_type=self.args.channel_scoring_type, + top_k_vocab=self.args.top_k_vocab, + lm_models=lm_models, + lm_dict=lm_dict, + lm_weight=getattr(self.args, 'lm_wt', 1), + normalize_lm_scores_by_tgt_len=getattr(self.args, 'normalize_lm_scores_by_tgt_len', False), + ) diff --git a/data/fairseq/examples/flores101/README.md b/data/fairseq/examples/flores101/README.md new file mode 100644 index 0000000000000000000000000000000000000000..635c13f40bd0ccab704735bc5c26ea0192ea98cd --- /dev/null +++ b/data/fairseq/examples/flores101/README.md @@ -0,0 +1,223 @@ +

+ +

+ +# Flores101: Large-Scale Multilingual Machine Translation + +## Introduction + +Baseline pretrained models for small and large tracks of WMT 21 Large-Scale Multilingual Machine Translation competition. + +Flores Task at WMT 21: http://www.statmt.org/wmt21/large-scale-multilingual-translation-task.html + +Flores announement blog post: https://ai.facebook.com/blog/flores-researchers-kick-off-multilingual-translation-challenge-at-wmt-and-call-for-compute-grants/ + + + +## Pretrained models + +Model | Num layers | Embed dimension | FFN dimension| Vocab Size | #params | Download +---|---|---|---|---|---|--- +`flores101_mm100_615M` | 12 | 1024 | 4096 | 256,000 | 615M | https://dl.fbaipublicfiles.com/flores101/pretrained_models/flores101_mm100_615M.tar.gz +`flores101_mm100_175M` | 6 | 512 | 2048 | 256,000 | 175M | https://dl.fbaipublicfiles.com/flores101/pretrained_models/flores101_mm100_175M.tar.gz + + +These models are trained similar to [M2M-100](https://arxiv.org/abs/2010.11125) with additional support for the languages that are part of the WMT Large-Scale Multilingual Machine Translation track. Full list of languages can be found at the bottom. + + +## Example Generation code + +### Download model, sentencepiece vocab + +```bash +fairseq=/path/to/fairseq +cd $fairseq + +# Download 615M param model. +wget https://dl.fbaipublicfiles.com/flores101/pretrained_models/flores101_mm100_615M.tar.gz + +# Extract +tar -xvzf flores101_mm100_615M.tar.gz +``` + +### Encode using our SentencePiece Model +Note: Install SentencePiece from [here](https://github.com/google/sentencepiece) + + +```bash +fairseq=/path/to/fairseq +cd $fairseq + +# Download example dataset From German to French +sacrebleu --echo src -l de-fr -t wmt19 | head -n 20 > raw_input.de-fr.de +sacrebleu --echo ref -l de-fr -t wmt19 | head -n 20 > raw_input.de-fr.fr + +for lang in de fr ; do + python scripts/spm_encode.py \ + --model flores101_mm100_615M/sentencepiece.bpe.model \ + --output_format=piece \ + --inputs=raw_input.de-fr.${lang} \ + --outputs=spm.de-fr.${lang} +done +``` + +### Binarization + +```bash +fairseq-preprocess \ + --source-lang de --target-lang fr \ + --testpref spm.de-fr \ + --thresholdsrc 0 --thresholdtgt 0 \ + --destdir data_bin \ + --srcdict flores101_mm100_615M/dict.txt --tgtdict flores101_mm100_615M/dict.txt +``` + +### Generation + + +```bash +fairseq-generate \ + data_bin \ + --batch-size 1 \ + --path flores101_mm100_615M/model.pt \ + --fixed-dictionary flores101_mm100_615M/dict.txt \ + -s de -t fr \ + --remove-bpe 'sentencepiece' \ + --beam 5 \ + --task translation_multi_simple_epoch \ + --lang-pairs flores101_mm100_615M/language_pairs.txt \ + --decoder-langtok --encoder-langtok src \ + --gen-subset test \ + --fp16 \ + --dataset-impl mmap \ + --distributed-world-size 1 --distributed-no-spawn +``` + +### Supported Languages and lang code + +Language | lang code +---|--- +Akrikaans | af +Amharic | am +Arabic | ar +Assamese | as +Asturian | ast +Aymara | ay +Azerbaijani | az +Bashkir | ba +Belarusian | be +Bulgarian | bg +Bengali | bn +Breton | br +Bosnian | bs +Catalan | ca +Cebuano | ceb +Chokwe | cjk +Czech | cs +Welsh | cy +Danish | da +German | de +Dyula| dyu +Greek | el +English | en +Spanish | es +Estonian | et +Persian | fa +Fulah | ff +Finnish | fi +French | fr +Western Frisian | fy +Irish | ga +Scottish Gaelic | gd +Galician | gl +Gujarati | gu +Hausa | ha +Hebrew | he +Hindi | hi +Croatian | hr +Haitian Creole | ht +Hungarian | hu +Armenian | hy +Indonesian | id +Igbo | ig +Iloko | ilo +Icelandic | is +Italian | it +Japanese | ja +Javanese | jv +Georgian | ka +Kachin | kac +Kamba | kam +Kabuverdianu | kea +Kongo | kg +Kazakh | kk +Central Khmer | km +Kimbundu | kmb +Northern Kurdish | kmr +Kannada | kn +Korean | ko +Kurdish | ku +Kyrgyz | ky +Luxembourgish | lb +Ganda | lg +Lingala | ln +Lao | lo +Lithuanian | lt +Luo | luo +Latvian | lv +Malagasy | mg +Maori | mi +Macedonian | mk +Malayalam | ml +Mongolian | mn +Marathi | mr +Malay | ms +Maltese | mt +Burmese | my +Nepali | ne +Dutch | nl +Norwegian | no +Northern Sotho | ns +Nyanja | ny +Occitan | oc +Oromo | om +Oriya | or +Punjabi | pa +Polish | pl +Pashto | ps +Portuguese | pt +Quechua | qu +Romanian | ro +Russian | ru +Sindhi | sd +Shan | shn +Sinhala | si +Slovak | sk +Slovenian | sl +Shona | sn +Somali | so +Albanian | sq +Serbian | sr +Swati | ss +Sundanese | su +Swedish | sv +Swahili | sw +Tamil | ta +Telugu | te +Tajik | tg +Thai | th +Tigrinya | ti +Tagalog | tl +Tswana | tn +Turkish | tr +Ukrainian | uk +Umbundu | umb +Urdu | ur +Uzbek | uz +Vietnamese | vi +Wolof | wo +Xhosa | xh +Yiddish | yi +Yoruba | yo +Chinese| zh +Zulu | zu diff --git a/data/fairseq/examples/flores101/flores_logo.png b/data/fairseq/examples/flores101/flores_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..d4d1455c6eab608ff5317ce885183cd213564273 Binary files /dev/null and b/data/fairseq/examples/flores101/flores_logo.png differ diff --git a/data/fairseq/examples/gottbert/README.md b/data/fairseq/examples/gottbert/README.md new file mode 100644 index 0000000000000000000000000000000000000000..1d58feb279a4a50222290546c3bb285d3cea98e6 --- /dev/null +++ b/data/fairseq/examples/gottbert/README.md @@ -0,0 +1,64 @@ +# GottBERT: a pure German language model + +## Introduction + +[GottBERT](http://arxiv.org/abs/2012.02110) is a pretrained language model trained on 145GB of German text based on RoBERTa. + +## Example usage + +### fairseq +##### Load GottBERT from torch.hub (PyTorch >= 1.1): +```python +import torch +gottbert = torch.hub.load('pytorch/fairseq', 'gottbert-base') +gottbert.eval() # disable dropout (or leave in train mode to finetune) +``` + +##### Load GottBERT (for PyTorch 1.0 or custom models): +```python +# Download gottbert model +wget https://dl.gottbert.de/fairseq/models/gottbert-base.tar.gz +tar -xzvf gottbert.tar.gz + +# Load the model in fairseq +from fairseq.models.roberta import GottbertModel +gottbert = GottbertModel.from_pretrained('/path/to/gottbert') +gottbert.eval() # disable dropout (or leave in train mode to finetune) +``` + +##### Filling masks: +```python +masked_line = 'Gott ist ! :)' +gottbert.fill_mask(masked_line, topk=3) +# [('Gott ist gut ! :)', 0.3642110526561737, ' gut'), +# ('Gott ist überall ! :)', 0.06009674072265625, ' überall'), +# ('Gott ist großartig ! :)', 0.0370681993663311, ' großartig')] +``` + +##### Extract features from GottBERT + +```python +# Extract the last layer's features +line = "Der erste Schluck aus dem Becher der Naturwissenschaft macht atheistisch , aber auf dem Grunde des Bechers wartet Gott !" +tokens = gottbert.encode(line) +last_layer_features = gottbert.extract_features(tokens) +assert last_layer_features.size() == torch.Size([1, 27, 768]) + +# Extract all layer's features (layer 0 is the embedding layer) +all_layers = gottbert.extract_features(tokens, return_all_hiddens=True) +assert len(all_layers) == 13 +assert torch.all(all_layers[-1] == last_layer_features) +``` +## Citation +If you use our work, please cite: + +```bibtex +@misc{scheible2020gottbert, + title={GottBERT: a pure German Language Model}, + author={Raphael Scheible and Fabian Thomczyk and Patric Tippmann and Victor Jaravine and Martin Boeker}, + year={2020}, + eprint={2012.02110}, + archivePrefix={arXiv}, + primaryClass={cs.CL} +} +``` diff --git a/data/fairseq/examples/hubert/README.md b/data/fairseq/examples/hubert/README.md new file mode 100644 index 0000000000000000000000000000000000000000..6695d819713b3e2077cf0fab30469b237e1cf1be --- /dev/null +++ b/data/fairseq/examples/hubert/README.md @@ -0,0 +1,116 @@ +# HuBERT + +## Pre-trained and fine-tuned (ASR) models +Model | Pretraining Data | Finetuning Dataset | Model | Quantizer +|---|---|---|---|--- +HuBERT Base (~95M params) | [Librispeech](http://www.openslr.org/12) 960 hr | No finetuning (Pretrained Model) | [download](https://dl.fbaipublicfiles.com/hubert/hubert_base_ls960.pt) | [L9 km500](https://dl.fbaipublicfiles.com/hubert/hubert_base_ls960_L9_km500.bin) +HuBERT Large (~316M params) | [Libri-Light](https://github.com/facebookresearch/libri-light) 60k hr | No finetuning (Pretrained Model) | [download](https://dl.fbaipublicfiles.com/hubert/hubert_large_ll60k.pt) +HuBERT Extra Large (~1B params) | [Libri-Light](https://github.com/facebookresearch/libri-light) 60k hr | No finetuning (Pretrained Model) | [download](https://dl.fbaipublicfiles.com/hubert/hubert_xtralarge_ll60k.pt) +HuBERT Large | [Libri-Light](https://github.com/facebookresearch/libri-light) 60k hr | [Librispeech](http://www.openslr.org/12) 960 hr | [download](https://dl.fbaipublicfiles.com/hubert/hubert_large_ll60k_finetune_ls960.pt) +HuBERT Extra Large | [Libri-Light](https://github.com/facebookresearch/libri-light) 60k hr | [Librispeech](http://www.openslr.org/12) 960 hr | [download](https://dl.fbaipublicfiles.com/hubert/hubert_xtralarge_ll60k_finetune_ls960.pt) + +## Load a model +``` +ckpt_path = "/path/to/the/checkpoint.pt" +models, cfg, task = fairseq.checkpoint_utils.load_model_ensemble_and_task([ckpt_path]) +model = models[0] +``` + +## Train a new model + +### Data preparation + +Follow the steps in `./simple_kmeans` to create: +- `{train,valid}.tsv` waveform list files +- `{train,valid}.km` frame-aligned pseudo label files. +- `dict.km.txt` a dummy dictionary +The `label_rate` is the same as the feature frame rate used for clustering, +which is 100Hz for MFCC features and 50Hz for HuBERT features by default. + +### Pre-train a HuBERT model + +Suppose `{train,valid}.tsv` are saved at `/path/to/data`, `{train,valid}.km` +are saved at `/path/to/labels`, and the label rate is 100Hz. + +To train a base model (12 layer transformer), run: +```sh +$ python fairseq_cli/hydra_train.py \ + --config-dir /path/to/fairseq-py/examples/hubert/config/pretrain \ + --config-name hubert_base_librispeech \ + task.data=/path/to/data task.label_dir=/path/to/labels task.labels='["km"]' model.label_rate=100 +``` + +### Fine-tune a HuBERT model with a CTC loss + +Suppose `{train,valid}.tsv` are saved at `/path/to/data`, and their +corresponding character transcripts `{train,valid}.ltr` are saved at +`/path/to/trans`. + +To fine-tune a pre-trained HuBERT model at `/path/to/checkpoint`, run +```sh +$ python fairseq_cli/hydra_train.py \ + --config-dir /path/to/fairseq-py/examples/hubert/config/finetune \ + --config-name base_10h \ + task.data=/path/to/data task.label_dir=/path/to/trans \ + model.w2v_path=/path/to/checkpoint +``` + +### Decode a HuBERT model + +Suppose the `test.tsv` and `test.ltr` are the waveform list and transcripts of +the split to be decoded, saved at `/path/to/data`, and the fine-tuned model is +saved at `/path/to/checkpoint`. We support three decoding modes: +- Viterbi decoding: greedy decoding without a language model +- KenLM decoding: decoding with an arpa-format KenLM n-gram language model +- Fairseq-LM deocding: decoding with a Fairseq neural language model + + +#### Viterbi decoding + +`task.normalize` needs to be consistent with the value used during fine-tuning. +Decoding results will be saved at +`/path/to/experiment/directory/decode/viterbi/test`. + +```sh +$ python examples/speech_recognition/new/infer.py \ + --config-dir /path/to/fairseq-py/examples/hubert/config/decode \ + --config-name infer_viterbi \ + task.data=/path/to/data \ + task.normalize=[true|false] \ + decoding.exp_dir=/path/to/experiment/directory \ + common_eval.path=/path/to/checkpoint + dataset.gen_subset=test \ +``` + +#### KenLM / Fairseq-LM decoding + +Suppose the pronunciation lexicon and the n-gram LM are saved at +`/path/to/lexicon` and `/path/to/arpa`, respectively. Decoding results will be +saved at `/path/to/experiment/directory/decode/kenlm/test`. + +```sh +$ python examples/speech_recognition/new/infer.py \ + --config-dir /path/to/fairseq-py/examples/hubert/config/decode \ + --config-name infer_kenlm \ + task.data=/path/to/data \ + task.normalize=[true|false] \ + decoding.exp_dir=/path/to/experiment/directory \ + common_eval.path=/path/to/checkpoint + dataset.gen_subset=test \ + decoding.decoder.lexicon=/path/to/lexicon \ + decoding.decoder.lmpath=/path/to/arpa +``` + +The command above uses the default decoding hyperparameter, which can be found +in `examples/speech_recognition/hydra/decoder.py`. These parameters can be +configured from the command line. For example, to search with a beam size of +500, we can append the command above with `decoding.decoder.beam=500`. +Important parameters include: +- decoding.decoder.beam +- decoding.decoder.beamthreshold +- decoding.decoder.lmweight +- decoding.decoder.wordscore +- decoding.decoder.silweight + +To decode with a Fairseq LM, use `--config-name infer_fsqlm` instead, and +change the path of lexicon and LM accordingly. diff --git a/data/fairseq/examples/hubert/config/decode/ax_sweep/ngram.yaml b/data/fairseq/examples/hubert/config/decode/ax_sweep/ngram.yaml new file mode 100644 index 0000000000000000000000000000000000000000..5a02df1f7da7eebfebe4018ef2758a716fbab646 --- /dev/null +++ b/data/fairseq/examples/hubert/config/decode/ax_sweep/ngram.yaml @@ -0,0 +1,33 @@ +# @package _global_ + +common_eval: + results_path: ${decoding.exp_dir}/decode/${decoding.decoder.name}_ax/${dataset.gen_subset} + +hydra: + sweeper: + ax_config: + max_trials: 60 + early_stop: + minimize: true + max_epochs_without_improvement: 10 + epsilon: 0.025 + experiment: + name: ${dataset.gen_subset} + objective_name: wer + minimize: true + parameter_constraints: null + outcome_constraints: null + status_quo: null + client: + verbose_logging: false + random_seed: null + params: + decoding.decoder.lmweight: + type: range + bounds: [0.0, 8.0] + decoding.decoder.wordscore: + type: range + bounds: [-5.0, 5.0] + decoding.decoder.silweight: + type: range + bounds: [-10.0, 0.0] diff --git a/data/fairseq/examples/hubert/config/decode/ax_sweep/transformer.yaml b/data/fairseq/examples/hubert/config/decode/ax_sweep/transformer.yaml new file mode 100644 index 0000000000000000000000000000000000000000..85ed3bd1a5a44871260f572786044c28f441add6 --- /dev/null +++ b/data/fairseq/examples/hubert/config/decode/ax_sweep/transformer.yaml @@ -0,0 +1,33 @@ +# @package _global_ + +common_eval: + results_path: ${decoding.exp_dir}/decode/${decoding.decoder.name}_ax/${dataset.gen_subset} + +hydra: + sweeper: + ax_config: + max_trials: 60 + early_stop: + minimize: true + max_epochs_without_improvement: 10 + epsilon: 0.025 + experiment: + name: ${dataset.gen_subset} + objective_name: wer + minimize: true + parameter_constraints: null + outcome_constraints: null + status_quo: null + client: + verbose_logging: false + random_seed: null + params: + decoding.decoder.lmweight: + type: range + bounds: [0.0, 4.0] + decoding.decoder.wordscore: + type: range + bounds: [-5.0, 5.0] + decoding.decoder.silweight: + type: range + bounds: [-8.0, 0.0] diff --git a/data/fairseq/examples/hubert/config/decode/infer_fsqlm.yaml b/data/fairseq/examples/hubert/config/decode/infer_fsqlm.yaml new file mode 100644 index 0000000000000000000000000000000000000000..026ad8db89a0673969a99fed6e1e84fc41fc7a1a --- /dev/null +++ b/data/fairseq/examples/hubert/config/decode/infer_fsqlm.yaml @@ -0,0 +1,36 @@ +# @package _group_ + +defaults: + - model: null + +hydra: + run: + dir: ${common_eval.results_path}/beam${decoding.beam}_th${decoding.beamthreshold}_lmw${decoding.lmweight}_wrd${decoding.wordscore}_sil${decoding.silweight} + sweep: + dir: ${common_eval.results_path} + subdir: beam${decoding.beam}_th${decoding.beamthreshold}_lmw${decoding.lmweight}_wrd${decoding.wordscore}_sil${decoding.silweight} + +task: + _name: hubert_pretraining + single_target: true + fine_tuning: true + data: ??? + normalize: ??? + +decoding: + type: fairseqlm + lexicon: ??? + lmpath: ??? + beamthreshold: 25 + beam: 500 + lmweight: 2 + wordscore: -1 + silweight: 0 + unique_wer_file: true +common_eval: + results_path: ??? + path: ??? + post_process: letter +dataset: + max_tokens: 1100000 + gen_subset: ??? diff --git a/data/fairseq/examples/hubert/config/decode/infer_kenlm.yaml b/data/fairseq/examples/hubert/config/decode/infer_kenlm.yaml new file mode 100644 index 0000000000000000000000000000000000000000..04642aeb6530133ab44e12e11e3d1661e3b9c32c --- /dev/null +++ b/data/fairseq/examples/hubert/config/decode/infer_kenlm.yaml @@ -0,0 +1,36 @@ +# @package _group_ + +defaults: + - model: null + +hydra: + run: + dir: ${common_eval.results_path}/beam${decoding.beam}_th${decoding.beamthreshold}_lmw${decoding.lmweight}_wrd${decoding.wordscore}_sil${decoding.silweight} + sweep: + dir: ${common_eval.results_path} + subdir: beam${decoding.beam}_th${decoding.beamthreshold}_lmw${decoding.lmweight}_wrd${decoding.wordscore}_sil${decoding.silweight} + +task: + _name: hubert_pretraining + single_target: true + fine_tuning: true + data: ??? + normalize: ??? + +decoding: + type: kenlm + lexicon: ??? + lmpath: ??? + beamthreshold: 100 + beam: 500 + lmweight: 2 + wordscore: -1 + silweight: 0 + unique_wer_file: true +common_eval: + results_path: ??? + path: ??? + post_process: letter +dataset: + max_tokens: 1100000 + gen_subset: ??? diff --git a/data/fairseq/examples/hubert/config/decode/infer_viterbi.yaml b/data/fairseq/examples/hubert/config/decode/infer_viterbi.yaml new file mode 100644 index 0000000000000000000000000000000000000000..4afc74c18ca890e1a20c6beabeb9059dd0f480f4 --- /dev/null +++ b/data/fairseq/examples/hubert/config/decode/infer_viterbi.yaml @@ -0,0 +1,29 @@ +# @package _group_ + +defaults: + - model: null + +hydra: + run: + dir: ${common_eval.results_path}/viterbi + sweep: + dir: ${common_eval.results_path} + subdir: viterbi + +task: + _name: hubert_pretraining + single_target: true + fine_tuning: true + data: ??? + normalize: ??? + +decoding: + type: viterbi + unique_wer_file: true +common_eval: + results_path: ??? + path: ??? + post_process: letter +dataset: + max_tokens: 1100000 + gen_subset: ??? diff --git a/data/fairseq/examples/hubert/config/decode/run/submitit_slurm.yaml b/data/fairseq/examples/hubert/config/decode/run/submitit_slurm.yaml new file mode 100644 index 0000000000000000000000000000000000000000..0b8065832ecacf9dd4fe4e99c87941e00fb3ef7f --- /dev/null +++ b/data/fairseq/examples/hubert/config/decode/run/submitit_slurm.yaml @@ -0,0 +1,17 @@ +# @package _global_ +hydra: + launcher: + cpus_per_task: ${distributed_training.distributed_world_size} + gpus_per_node: ${distributed_training.distributed_world_size} + tasks_per_node: ${hydra.launcher.gpus_per_node} + nodes: 1 + mem_gb: 200 + timeout_min: 4320 + max_num_timeout: 50 + name: ${hydra.job.config_name} + submitit_folder: ${hydra.sweep.dir}/submitit + +distributed_training: + distributed_world_size: 1 + distributed_no_spawn: true + distributed_port: 29761 diff --git a/data/fairseq/examples/hubert/config/decode/run/submitit_slurm_8gpu.yaml b/data/fairseq/examples/hubert/config/decode/run/submitit_slurm_8gpu.yaml new file mode 100644 index 0000000000000000000000000000000000000000..2f669f376312dbfe4611cc08f4996a314155fb87 --- /dev/null +++ b/data/fairseq/examples/hubert/config/decode/run/submitit_slurm_8gpu.yaml @@ -0,0 +1,17 @@ +# @package _global_ +hydra: + launcher: + cpus_per_task: ${distributed_training.distributed_world_size} + gpus_per_node: ${distributed_training.distributed_world_size} + tasks_per_node: ${hydra.launcher.gpus_per_node} + nodes: 1 + mem_gb: 200 + timeout_min: 4320 + max_num_timeout: 50 + name: ${hydra.job.config_name} + submitit_folder: ${hydra.sweep.dir}/submitit + +distributed_training: + distributed_world_size: 8 + distributed_no_spawn: true + distributed_port: 29761 diff --git a/data/fairseq/examples/hubert/config/finetune/base_10h.yaml b/data/fairseq/examples/hubert/config/finetune/base_10h.yaml new file mode 100644 index 0000000000000000000000000000000000000000..a22c7c0347f792221f209bcfba7ba380a69f90a8 --- /dev/null +++ b/data/fairseq/examples/hubert/config/finetune/base_10h.yaml @@ -0,0 +1,100 @@ +# @package _group_ + +common: + fp16: true + log_format: json + log_interval: 200 + tensorboard_logdir: tblog + seed: 1337 + +checkpoint: + save_interval: 5 + keep_interval_updates: 1 + no_epoch_checkpoints: true + best_checkpoint_metric: wer + +distributed_training: + ddp_backend: c10d + find_unused_parameters: true + distributed_world_size: 1 + distributed_port: 29671 + nprocs_per_node: 8 + +task: + _name: hubert_pretraining + data: ??? + fine_tuning: true + label_dir: ??? + normalize: false # must be consistent with pre-training + labels: ["ltr"] + single_target: true + +dataset: + num_workers: 0 + max_tokens: 3200000 + validate_after_updates: ${model.freeze_finetune_updates} + validate_interval: 5 + train_subset: train + valid_subset: valid + +criterion: + _name: ctc + zero_infinity: true + +optimization: + max_update: 25000 + lr: [2e-5] + sentence_avg: true + update_freq: [1] + +optimizer: + _name: adam + adam_betas: (0.9,0.98) + adam_eps: 1e-08 + +lr_scheduler: + _name: tri_stage + warmup_steps: 8000 + hold_steps: 0 + decay_steps: 72000 + final_lr_scale: 0.05 + +model: + _name: hubert_ctc + w2v_path: ??? + apply_mask: true + mask_selection: static + mask_length: 10 + mask_other: 0 + mask_prob: 0.75 + mask_channel_selection: static + mask_channel_length: 64 + mask_channel_other: 0 + mask_channel_prob: 0.5 + layerdrop: 0.1 + dropout: 0.0 + activation_dropout: 0.1 + attention_dropout: 0.0 + feature_grad_mult: 0.0 + freeze_finetune_updates: 10000 + +hydra: + job: + config: + override_dirname: + kv_sep: '-' + item_sep: '__' + exclude_keys: + - run + - task.data + - task.label_dir + - model.w2v_path + - dataset.train_subset + - dataset.valid_subset + - criterion.wer_kenlm_model + - criterion.wer_lexicon + run: + dir: ??? + sweep: + dir: ??? + subdir: ${hydra.job.config_name}__${hydra.job.override_dirname} diff --git a/data/fairseq/examples/hubert/config/finetune/ckpt/it1.yaml b/data/fairseq/examples/hubert/config/finetune/ckpt/it1.yaml new file mode 100644 index 0000000000000000000000000000000000000000..2af96b3f72746f85feb13e7efcbdab6602b293de --- /dev/null +++ b/data/fairseq/examples/hubert/config/finetune/ckpt/it1.yaml @@ -0,0 +1,7 @@ +# @package _global_ + +task: + normalize: false + +model: + w2v_path: /checkpoint/wnhsu/w2v/hubert_final/iter1/hubert.km.randcrop.pmw1_0.puw0_0.grpnorm.ml10.mp0_8.untie.mxsz250000.ufreq1.maxtok1400000.MU400k.s1337.ngpu32/checkpoint_last.pt diff --git a/data/fairseq/examples/hubert/config/finetune/lm/ls_4gram.yaml b/data/fairseq/examples/hubert/config/finetune/lm/ls_4gram.yaml new file mode 100644 index 0000000000000000000000000000000000000000..8c7728ad29965d3cf18605808a893bc442afd56b --- /dev/null +++ b/data/fairseq/examples/hubert/config/finetune/lm/ls_4gram.yaml @@ -0,0 +1,7 @@ +# @package _global_ + +criterion: + wer_kenlm_model: /checkpoint/abdo/old_checkpoint02/datasets/librispeech/4-gram.bin + wer_lexicon: /checkpoint/abdo/old_checkpoint02/datasets/librispeech/10h/raw/lexicon_ltr.lst + wer_lm_weight: 2.0 + wer_word_score: -1.0 diff --git a/data/fairseq/examples/hubert/config/finetune/run/submitit_reg.yaml b/data/fairseq/examples/hubert/config/finetune/run/submitit_reg.yaml new file mode 100644 index 0000000000000000000000000000000000000000..27509503e7b306c07742fbed2fc5726d001bb7df --- /dev/null +++ b/data/fairseq/examples/hubert/config/finetune/run/submitit_reg.yaml @@ -0,0 +1,20 @@ +# @package _global_ + +hydra: + launcher: + cpus_per_task: 8 + gpus_per_node: 8 + tasks_per_node: ${hydra.launcher.gpus_per_node} + nodes: 1 + comment: null + mem_gb: 384 + timeout_min: 4320 + max_num_timeout: 100 + constraint: volta32gb + name: ${hydra.job.config_name}/${hydra.job.override_dirname} + submitit_folder: ${hydra.sweep.dir}/submitit/%j + +distributed_training: + distributed_world_size: 8 + distributed_port: 29671 + nprocs_per_node: 8 diff --git a/data/fairseq/examples/hubert/config/pretrain/data/iter1.yaml b/data/fairseq/examples/hubert/config/pretrain/data/iter1.yaml new file mode 100644 index 0000000000000000000000000000000000000000..0a1b65d802c83128c53f32b21807fa5e51da6cc9 --- /dev/null +++ b/data/fairseq/examples/hubert/config/pretrain/data/iter1.yaml @@ -0,0 +1,8 @@ +# @package _global_ + +task: + label_dir: ??? + labels: ["km"] + +model: + label_rate: 100 diff --git a/data/fairseq/examples/hubert/config/pretrain/data/iter2.yaml b/data/fairseq/examples/hubert/config/pretrain/data/iter2.yaml new file mode 100644 index 0000000000000000000000000000000000000000..2d4bfe61cc638af9de48e92c58994e435fba2abf --- /dev/null +++ b/data/fairseq/examples/hubert/config/pretrain/data/iter2.yaml @@ -0,0 +1,8 @@ +# @package _global_ + +task: + label_dir: ??? + labels: ["km"] + +model: + label_rate: 50 diff --git a/data/fairseq/examples/hubert/config/pretrain/hubert_base_librispeech.yaml b/data/fairseq/examples/hubert/config/pretrain/hubert_base_librispeech.yaml new file mode 100644 index 0000000000000000000000000000000000000000..bd84461a163866f622b01bf6d36b4de6215f3d97 --- /dev/null +++ b/data/fairseq/examples/hubert/config/pretrain/hubert_base_librispeech.yaml @@ -0,0 +1,97 @@ +# @package _group_ + +common: + fp16: true + log_format: json + log_interval: 200 + seed: 1337 + tensorboard_logdir: tblog + +checkpoint: + save_interval_updates: 25000 + keep_interval_updates: 1 + no_epoch_checkpoints: true + + +distributed_training: + ddp_backend: no_c10d + distributed_backend: 'nccl' + distributed_world_size: 32 + distributed_port: 29671 + nprocs_per_node: 8 + find_unused_parameters: true + +task: + _name: hubert_pretraining + data: ??? + label_dir: ??? + labels: ??? + label_rate: ${model.label_rate} + sample_rate: 16000 + max_sample_size: 250000 + min_sample_size: 32000 + pad_audio: false + random_crop: true + normalize: false # must be consistent with extractor + +dataset: + num_workers: 6 + max_tokens: 1400000 + skip_invalid_size_inputs_valid_test: true + validate_interval: 5 + validate_interval_updates: 10000 + +criterion: + _name: hubert + pred_masked_weight: 1.0 + pred_nomask_weight: 0.0 + loss_weights: [10,] + +optimization: + max_update: 400000 + lr: [0.0005] + clip_norm: 10.0 + +optimizer: + _name: adam + adam_betas: (0.9,0.98) + adam_eps: 1e-06 + weight_decay: 0.01 + +lr_scheduler: + _name: polynomial_decay + warmup_updates: 32000 + +model: + _name: hubert + label_rate: ??? + skip_masked: false + skip_nomask: false + mask_prob: 0.80 + extractor_mode: default + conv_feature_layers: '[(512,10,5)] + [(512,3,2)] * 4 + [(512,2,2)] * 2' + final_dim: 256 + encoder_layerdrop: 0.05 + dropout_input: 0.1 + dropout_features: 0.1 + dropout: 0.1 + attention_dropout: 0.1 + feature_grad_mult: 0.1 + untie_final_proj: true + activation_dropout: 0.0 + +hydra: + job: + config: + override_dirname: + kv_sep: '-' + item_sep: '__' + exclude_keys: + - run + - task.data + - task.label_dir + run: + dir: ??? + sweep: + dir: ??? + subdir: ${hydra.job.config_name}__${hydra.job.override_dirname} diff --git a/data/fairseq/examples/hubert/config/pretrain/hubert_large_librivox.yaml b/data/fairseq/examples/hubert/config/pretrain/hubert_large_librivox.yaml new file mode 100644 index 0000000000000000000000000000000000000000..a5192b5f29b53aa8391a0ab67b6238c0d0b4985e --- /dev/null +++ b/data/fairseq/examples/hubert/config/pretrain/hubert_large_librivox.yaml @@ -0,0 +1,101 @@ +# @package _group_ + +common: + fp16: true + log_format: json + log_interval: 200 + seed: 1337 + tensorboard_logdir: tblog + +checkpoint: + save_interval_updates: 25000 + keep_interval_updates: 1 + no_epoch_checkpoints: true + + +distributed_training: + ddp_backend: no_c10d + distributed_backend: 'nccl' + distributed_world_size: 128 + distributed_port: 29671 + nprocs_per_node: 8 + find_unused_parameters: true + +task: + _name: hubert_pretraining + data: ??? + label_dir: ??? + labels: ??? + label_rate: ${model.label_rate} + sample_rate: 16000 + max_sample_size: 250000 + min_sample_size: 32000 + pad_audio: false + random_crop: true + normalize: true # must be consistent with extractor + +dataset: + num_workers: 6 + max_tokens: 900000 + skip_invalid_size_inputs_valid_test: true + validate_interval: 5 + validate_interval_updates: 10000 + +criterion: + _name: hubert + pred_masked_weight: 1.0 + pred_nomask_weight: 0.0 + loss_weights: [10,] + +optimization: + max_update: 400000 + lr: [0.0015] + clip_norm: 1.0 + +optimizer: + _name: adam + adam_betas: (0.9,0.98) + adam_eps: 1e-06 + weight_decay: 0.01 + +lr_scheduler: + _name: polynomial_decay + warmup_updates: 32000 + +model: + _name: hubert + label_rate: ??? + encoder_layers: 24 + encoder_embed_dim: 1024 + encoder_ffn_embed_dim: 4096 + encoder_attention_heads: 16 + final_dim: 768 + skip_masked: false + skip_nomask: false + mask_prob: 0.80 + extractor_mode: layer_norm + conv_feature_layers: '[(512,10,5)] + [(512,3,2)] * 4 + [(512,2,2)] * 2' + encoder_layerdrop: 0.0 + dropout_input: 0.0 + dropout_features: 0.0 + dropout: 0.0 + attention_dropout: 0.0 + layer_norm_first: true + feature_grad_mult: 1.0 + untie_final_proj: true + activation_dropout: 0.0 + +hydra: + job: + config: + override_dirname: + kv_sep: '-' + item_sep: '__' + exclude_keys: + - run + - task.data + run: + dir: /checkpoint/wnhsu/w2v/hubert_final/hydra_pt + sweep: + dir: /checkpoint/wnhsu/w2v/hubert_final/hydra_pt + subdir: ${hydra.job.config_name}__${hydra.job.override_dirname} diff --git a/data/fairseq/examples/hubert/config/pretrain/hubert_xlarge_librivox.yaml b/data/fairseq/examples/hubert/config/pretrain/hubert_xlarge_librivox.yaml new file mode 100644 index 0000000000000000000000000000000000000000..34e8f2bfb93863db122f694785b80857713ceb05 --- /dev/null +++ b/data/fairseq/examples/hubert/config/pretrain/hubert_xlarge_librivox.yaml @@ -0,0 +1,101 @@ +# @package _group_ + +common: + fp16: true + log_format: json + log_interval: 200 + seed: 1337 + tensorboard_logdir: tblog + +checkpoint: + save_interval_updates: 25000 + keep_interval_updates: 1 + no_epoch_checkpoints: true + + +distributed_training: + ddp_backend: no_c10d + distributed_backend: 'nccl' + distributed_world_size: 256 + distributed_port: 29671 + nprocs_per_node: 8 + find_unused_parameters: true + +task: + _name: hubert_pretraining + data: ??? + label_dir: ??? + labels: ??? + label_rate: ${model.label_rate} + sample_rate: 16000 + max_sample_size: 250000 + min_sample_size: 32000 + pad_audio: false + random_crop: true + normalize: true # must be consistent with extractor + +dataset: + num_workers: 6 + max_tokens: 360000 + skip_invalid_size_inputs_valid_test: true + validate_interval: 5 + validate_interval_updates: 10000 + +criterion: + _name: hubert + pred_masked_weight: 1.0 + pred_nomask_weight: 0.0 + loss_weights: [10,] + +optimization: + max_update: 400000 + lr: [0.003] + clip_norm: 1.0 + +optimizer: + _name: adam + adam_betas: (0.9,0.98) + adam_eps: 1e-06 + weight_decay: 0.01 + +lr_scheduler: + _name: polynomial_decay + warmup_updates: 32000 + +model: + _name: hubert + label_rate: ??? + encoder_layers: 48 + encoder_embed_dim: 1280 + encoder_ffn_embed_dim: 5120 + encoder_attention_heads: 16 + final_dim: 1024 + skip_masked: false + skip_nomask: false + mask_prob: 0.80 + extractor_mode: layer_norm + conv_feature_layers: '[(512,10,5)] + [(512,3,2)] * 4 + [(512,2,2)] * 2' + encoder_layerdrop: 0.0 + dropout_input: 0.0 + dropout_features: 0.0 + dropout: 0.0 + attention_dropout: 0.0 + layer_norm_first: true + feature_grad_mult: 1.0 + untie_final_proj: true + activation_dropout: 0.0 + +hydra: + job: + config: + override_dirname: + kv_sep: '-' + item_sep: '__' + exclude_keys: + - run + - task.data + run: + dir: /checkpoint/wnhsu/w2v/hubert_final/hydra_pt + sweep: + dir: /checkpoint/wnhsu/w2v/hubert_final/hydra_pt + subdir: ${hydra.job.config_name}__${hydra.job.override_dirname} diff --git a/data/fairseq/examples/hubert/config/pretrain/run/submitit_reg.yaml b/data/fairseq/examples/hubert/config/pretrain/run/submitit_reg.yaml new file mode 100644 index 0000000000000000000000000000000000000000..46c979cd2835fe026b0a532a54533904d1001e54 --- /dev/null +++ b/data/fairseq/examples/hubert/config/pretrain/run/submitit_reg.yaml @@ -0,0 +1,20 @@ +# @package _global_ + +hydra: + launcher: + cpus_per_task: 8 + gpus_per_node: 8 + tasks_per_node: ${hydra.launcher.gpus_per_node} + nodes: 4 + comment: null + mem_gb: 384 + timeout_min: 4320 + max_num_timeout: 100 + constraint: volta32gb + name: ${hydra.job.config_name}/${hydra.job.override_dirname} + submitit_folder: ${hydra.sweep.dir}/submitit/%j + +distributed_training: + distributed_world_size: 32 + distributed_port: 29671 + nprocs_per_node: 8 diff --git a/data/fairseq/examples/hubert/measure_teacher_quality.py b/data/fairseq/examples/hubert/measure_teacher_quality.py new file mode 100644 index 0000000000000000000000000000000000000000..92279b2214bb2ba4a99aea92098907ef4f55821b --- /dev/null +++ b/data/fairseq/examples/hubert/measure_teacher_quality.py @@ -0,0 +1,241 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import numpy as np +import os.path as op +import re +from tabulate import tabulate +from collections import Counter + + +def comp_purity(p_xy, axis): + max_p = p_xy.max(axis=axis) + marg_p = p_xy.sum(axis=axis) + indv_pur = max_p / marg_p + aggr_pur = max_p.sum() + return indv_pur, aggr_pur + + +def comp_entropy(p): + return (-p * np.log(p + 1e-8)).sum() + + +def comp_norm_mutual_info(p_xy): + p_x = p_xy.sum(axis=1, keepdims=True) + p_y = p_xy.sum(axis=0, keepdims=True) + pmi = np.log(p_xy / np.matmul(p_x, p_y) + 1e-8) + mi = (p_xy * pmi).sum() + h_x = comp_entropy(p_x) + h_y = comp_entropy(p_y) + return mi, mi / h_x, mi / h_y, h_x, h_y + + +def pad(labs, n): + if n == 0: + return np.array(labs) + return np.concatenate([[labs[0]] * n, labs, [labs[-1]] * n]) + + +def comp_avg_seg_dur(labs_list): + n_frms = 0 + n_segs = 0 + for labs in labs_list: + labs = np.array(labs) + edges = np.zeros(len(labs)).astype(bool) + edges[0] = True + edges[1:] = labs[1:] != labs[:-1] + n_frms += len(edges) + n_segs += edges.astype(int).sum() + return n_frms / n_segs + + +def comp_joint_prob(uid2refs, uid2hyps): + """ + Args: + pad: padding for spliced-feature derived labels + """ + cnts = Counter() + skipped = [] + abs_frmdiff = 0 + for uid in uid2refs: + if uid not in uid2hyps: + skipped.append(uid) + continue + refs = uid2refs[uid] + hyps = uid2hyps[uid] + abs_frmdiff += abs(len(refs) - len(hyps)) + min_len = min(len(refs), len(hyps)) + refs = refs[:min_len] + hyps = hyps[:min_len] + cnts.update(zip(refs, hyps)) + tot = sum(cnts.values()) + + ref_set = sorted({ref for ref, _ in cnts.keys()}) + hyp_set = sorted({hyp for _, hyp in cnts.keys()}) + ref2pid = dict(zip(ref_set, range(len(ref_set)))) + hyp2lid = dict(zip(hyp_set, range(len(hyp_set)))) + # print(hyp_set) + p_xy = np.zeros((len(ref2pid), len(hyp2lid)), dtype=float) + for (ref, hyp), cnt in cnts.items(): + p_xy[ref2pid[ref], hyp2lid[hyp]] = cnt + p_xy /= p_xy.sum() + return p_xy, ref2pid, hyp2lid, tot, abs_frmdiff, skipped + + +def read_phn(tsv_path, rm_stress=True): + uid2phns = {} + with open(tsv_path) as f: + for line in f: + uid, phns = line.rstrip().split("\t") + phns = phns.split(",") + if rm_stress: + phns = [re.sub("[0-9]", "", phn) for phn in phns] + uid2phns[uid] = phns + return uid2phns + + +def read_lab(tsv_path, lab_path, pad_len=0, upsample=1): + """ + tsv is needed to retrieve the uids for the labels + """ + with open(tsv_path) as f: + f.readline() + uids = [op.splitext(op.basename(line.rstrip().split()[0]))[0] for line in f] + with open(lab_path) as f: + labs_list = [pad(line.rstrip().split(), pad_len).repeat(upsample) for line in f] + assert len(uids) == len(labs_list) + return dict(zip(uids, labs_list)) + + +def main_lab_lab( + tsv_dir, + lab_dir, + lab_name, + lab_sets, + ref_dir, + ref_name, + pad_len=0, + upsample=1, + verbose=False, +): + # assume tsv_dir is the same for both the reference and the hypotheses + tsv_dir = lab_dir if tsv_dir is None else tsv_dir + + uid2refs = {} + for s in lab_sets: + uid2refs.update(read_lab(f"{tsv_dir}/{s}.tsv", f"{ref_dir}/{s}.{ref_name}")) + + uid2hyps = {} + for s in lab_sets: + uid2hyps.update( + read_lab( + f"{tsv_dir}/{s}.tsv", f"{lab_dir}/{s}.{lab_name}", pad_len, upsample + ) + ) + _main(uid2refs, uid2hyps, verbose) + + +def main_phn_lab( + tsv_dir, + lab_dir, + lab_name, + lab_sets, + phn_dir, + phn_sets, + pad_len=0, + upsample=1, + verbose=False, +): + uid2refs = {} + for s in phn_sets: + uid2refs.update(read_phn(f"{phn_dir}/{s}.tsv")) + + uid2hyps = {} + tsv_dir = lab_dir if tsv_dir is None else tsv_dir + for s in lab_sets: + uid2hyps.update( + read_lab( + f"{tsv_dir}/{s}.tsv", f"{lab_dir}/{s}.{lab_name}", pad_len, upsample + ) + ) + _main(uid2refs, uid2hyps, verbose) + + +def _main(uid2refs, uid2hyps, verbose): + (p_xy, ref2pid, hyp2lid, tot, frmdiff, skipped) = comp_joint_prob( + uid2refs, uid2hyps + ) + ref_pur_by_hyp, ref_pur = comp_purity(p_xy, axis=0) + hyp_pur_by_ref, hyp_pur = comp_purity(p_xy, axis=1) + (mi, mi_norm_by_ref, mi_norm_by_hyp, h_ref, h_hyp) = comp_norm_mutual_info(p_xy) + outputs = { + "ref pur": ref_pur, + "hyp pur": hyp_pur, + "H(ref)": h_ref, + "H(hyp)": h_hyp, + "MI": mi, + "MI/H(ref)": mi_norm_by_ref, + "ref segL": comp_avg_seg_dur(uid2refs.values()), + "hyp segL": comp_avg_seg_dur(uid2hyps.values()), + "p_xy shape": p_xy.shape, + "frm tot": tot, + "frm diff": frmdiff, + "utt tot": len(uid2refs), + "utt miss": len(skipped), + } + print(tabulate([outputs.values()], outputs.keys(), floatfmt=".4f")) + + +if __name__ == "__main__": + """ + compute quality of labels with respect to phone or another labels if set + """ + import argparse + + parser = argparse.ArgumentParser() + parser.add_argument("tsv_dir") + parser.add_argument("lab_dir") + parser.add_argument("lab_name") + parser.add_argument("--lab_sets", default=["valid"], type=str, nargs="+") + parser.add_argument( + "--phn_dir", + default="/checkpoint/wnhsu/data/librispeech/960h/fa/raw_phn/phone_frame_align_v1", + ) + parser.add_argument( + "--phn_sets", default=["dev-clean", "dev-other"], type=str, nargs="+" + ) + parser.add_argument("--pad_len", default=0, type=int, help="padding for hypotheses") + parser.add_argument( + "--upsample", default=1, type=int, help="upsample factor for hypotheses" + ) + parser.add_argument("--ref_lab_dir", default="") + parser.add_argument("--ref_lab_name", default="") + parser.add_argument("--verbose", action="store_true") + args = parser.parse_args() + + if args.ref_lab_dir and args.ref_lab_name: + main_lab_lab( + args.tsv_dir, + args.lab_dir, + args.lab_name, + args.lab_sets, + args.ref_lab_dir, + args.ref_lab_name, + args.pad_len, + args.upsample, + args.verbose, + ) + else: + main_phn_lab( + args.tsv_dir, + args.lab_dir, + args.lab_name, + args.lab_sets, + args.phn_dir, + args.phn_sets, + args.pad_len, + args.upsample, + args.verbose, + ) diff --git a/data/fairseq/examples/hubert/simple_kmeans/README.md b/data/fairseq/examples/hubert/simple_kmeans/README.md new file mode 100644 index 0000000000000000000000000000000000000000..847475c23f8a6a47bb25cba83466ddd9eba167b8 --- /dev/null +++ b/data/fairseq/examples/hubert/simple_kmeans/README.md @@ -0,0 +1,80 @@ +# Sharded Feature Extraction and K-means Application + +This folder contains scripts for preparing HUBERT labels from tsv files, the +steps are: +1. feature extraction +2. k-means clustering +3. k-means application + + +## Data preparation + +`*.tsv` files contains a list of audio, where each line is the root, and +following lines are the subpath for each audio: +``` + + + +... +``` + + +## Feature extraction + +### MFCC feature +Suppose the tsv file is at `${tsv_dir}/${split}.tsv`. To extract 39-D +mfcc+delta+ddelta features for the 1st iteration HUBERT training, run: +```sh +python dump_mfcc_feature.py ${tsv_dir} ${split} ${nshard} ${rank} ${feat_dir} +``` +This would shard the tsv file into `${nshard}` and extract features for the +`${rank}`-th shard, where rank is an integer in `[0, nshard-1]`. Features would +be saved at `${feat_dir}/${split}_${rank}_${nshard}.{npy,len}`. + + +### HUBERT feature +To extract features from the `${layer}`-th transformer layer of a trained +HUBERT model saved at `${ckpt_path}`, run: +```sh +python dump_hubert_feature.py ${tsv_dir} ${split} ${ckpt_path} ${layer} ${nshard} ${rank} ${feat_dir} +``` +Features would also be saved at `${feat_dir}/${split}_${rank}_${nshard}.{npy,len}`. + +- if out-of-memory, decrease the chunk size with `--max_chunk` + + +## K-means clustering +To fit a k-means model with `${n_clusters}` clusters on 10% of the `${split}` data, run +```sh +python learn_kmeans.py ${feat_dir} ${split} ${nshard} ${km_path} ${n_cluster} --percent 0.1 +``` +This saves the k-means model to `${km_path}`. + +- set `--precent -1` to use all data +- more kmeans options can be found with `-h` flag + + +## K-means application +To apply a trained k-means model `${km_path}` to obtain labels for `${split}`, run +```sh +python dump_km_label.py ${feat_dir} ${split} ${km_path} ${nshard} ${rank} ${lab_dir} +``` +This would extract labels for the `${rank}`-th shard out of `${nshard}` shards +and dump them to `${lab_dir}/${split}_${rank}_${shard}.km` + + +Finally, merge shards for `${split}` by running +```sh +for rank in $(seq 0 $((nshard - 1))); do + cat $lab_dir/${split}_${rank}_${nshard}.km +done > $lab_dir/${split}.km +``` + + +## Create a dummy dict +To create a dummy dictionary, run +```sh +for x in $(seq 0 $((n_clusters - 1))); do + echo "$x 1" +done >> $lab_dir/dict.km.txt +``` diff --git a/data/fairseq/examples/hubert/simple_kmeans/dump_hubert_feature.py b/data/fairseq/examples/hubert/simple_kmeans/dump_hubert_feature.py new file mode 100644 index 0000000000000000000000000000000000000000..7ea4ea0aa93046a133722511311a2735796cefeb --- /dev/null +++ b/data/fairseq/examples/hubert/simple_kmeans/dump_hubert_feature.py @@ -0,0 +1,93 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import logging +import os +import sys + +import fairseq +import soundfile as sf +import torch +import torch.nn.functional as F + +from feature_utils import get_path_iterator, dump_feature +from fairseq.data.audio.audio_utils import get_features_or_waveform + + +logging.basicConfig( + format="%(asctime)s | %(levelname)s | %(name)s | %(message)s", + datefmt="%Y-%m-%d %H:%M:%S", + level=os.environ.get("LOGLEVEL", "INFO").upper(), + stream=sys.stdout, +) +logger = logging.getLogger("dump_hubert_feature") + + +class HubertFeatureReader(object): + def __init__(self, ckpt_path, layer, max_chunk=1600000): + ( + model, + cfg, + task, + ) = fairseq.checkpoint_utils.load_model_ensemble_and_task([ckpt_path]) + self.model = model[0].eval().cuda() + self.task = task + self.layer = layer + self.max_chunk = max_chunk + logger.info(f"TASK CONFIG:\n{self.task.cfg}") + logger.info(f" max_chunk = {self.max_chunk}") + + def read_audio(self, path, ref_len=None): + wav = get_features_or_waveform(path, need_waveform=True, use_sample_rate=self.task.cfg.sample_rate) + if wav.ndim == 2: + wav = wav.mean(-1) + assert wav.ndim == 1, wav.ndim + if ref_len is not None and abs(ref_len - len(wav)) > 160: + logging.warning(f"ref {ref_len} != read {len(wav)} ({path})") + return wav + + def get_feats(self, path, ref_len=None): + x = self.read_audio(path, ref_len=ref_len) + with torch.no_grad(): + x = torch.from_numpy(x).float().cuda() + if self.task.cfg.normalize: + x = F.layer_norm(x, x.shape) + x = x.view(1, -1) + + feat = [] + for start in range(0, x.size(1), self.max_chunk): + x_chunk = x[:, start : start + self.max_chunk] + feat_chunk, _ = self.model.extract_features( + source=x_chunk, + padding_mask=None, + mask=False, + output_layer=self.layer, + ) + feat.append(feat_chunk) + return torch.cat(feat, 1).squeeze(0) + + +def main(tsv_dir, split, ckpt_path, layer, nshard, rank, feat_dir, max_chunk): + reader = HubertFeatureReader(ckpt_path, layer, max_chunk) + generator, num = get_path_iterator(f"{tsv_dir}/{split}.tsv", nshard, rank) + dump_feature(reader, generator, num, split, nshard, rank, feat_dir) + + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser() + parser.add_argument("tsv_dir") + parser.add_argument("split") + parser.add_argument("ckpt_path") + parser.add_argument("layer", type=int) + parser.add_argument("nshard", type=int) + parser.add_argument("rank", type=int) + parser.add_argument("feat_dir") + parser.add_argument("--max_chunk", type=int, default=1600000) + args = parser.parse_args() + logger.info(args) + + main(**vars(args)) diff --git a/data/fairseq/examples/hubert/simple_kmeans/dump_hubert_feature_s2t.py b/data/fairseq/examples/hubert/simple_kmeans/dump_hubert_feature_s2t.py new file mode 100644 index 0000000000000000000000000000000000000000..941bc1b675459b800b7e006f2ff9c2305c0dd8e8 --- /dev/null +++ b/data/fairseq/examples/hubert/simple_kmeans/dump_hubert_feature_s2t.py @@ -0,0 +1,95 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import csv +import io +import logging +import os +import os.path as op +import sys + +from dump_hubert_feature import HubertFeatureReader +from feature_utils import get_shard_range, dump_feature +from fairseq.data.audio.audio_utils import get_features_or_waveform + + +logging.basicConfig( + format="%(asctime)s | %(levelname)s | %(name)s | %(message)s", + datefmt="%Y-%m-%d %H:%M:%S", + level=os.environ.get("LOGLEVEL", "INFO").upper(), + stream=sys.stdout, +) +logger = logging.getLogger("dump_hubert_feature_s2t") + + +class HubertFeatureReaderS2T(HubertFeatureReader): + def read_audio(self, path, ref_len=None): + wav = get_features_or_waveform( + path, need_waveform=True, use_sample_rate=self.task.cfg.sample_rate + ) + if wav.ndim == 2: + wav = wav.mean(-1) + assert wav.ndim == 1, wav.ndim + if ref_len is not None and abs(ref_len - len(wav)) > 160: + logging.warning(f"ref {ref_len} != read {len(wav)} ({path})") + return wav + + +def get_path_iterator(root, tsv, nshard, rank, audio_col_name): + with open(tsv) as f: + reader = csv.DictReader( + f, + delimiter="\t", + quotechar=None, + doublequote=False, + lineterminator="\n", + quoting=csv.QUOTE_NONE, + ) + subpaths = [op.join(root, e[audio_col_name]) for e in reader] + start, end = get_shard_range(len(subpaths), nshard, rank) + subpaths = subpaths[start:end] + + def iterate(): + for subpath in subpaths: + yield op.join(root, subpath), None + + return iterate, len(subpaths) + + +def main( + root, + tsv_path, + ckpt_path, + layer, + nshard, + rank, + feat_dir, + split, + max_chunk, + audio_col_name, +): + reader = HubertFeatureReaderS2T(ckpt_path, layer, max_chunk) + generator, num = get_path_iterator(root, tsv_path, nshard, rank, audio_col_name) + dump_feature(reader, generator, num, split, nshard, rank, feat_dir) + + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser() + parser.add_argument("root") + parser.add_argument("tsv_path") + parser.add_argument("ckpt_path") + parser.add_argument("layer", type=int) + parser.add_argument("nshard", type=int) + parser.add_argument("rank", type=int) + parser.add_argument("feat_dir") + parser.add_argument("split") + parser.add_argument("--audio_col_name", type=str, default="audio") + parser.add_argument("--max_chunk", type=int, default=1600000) + args = parser.parse_args() + logger.info(args) + + main(**vars(args)) diff --git a/data/fairseq/examples/hubert/simple_kmeans/dump_km_label.py b/data/fairseq/examples/hubert/simple_kmeans/dump_km_label.py new file mode 100644 index 0000000000000000000000000000000000000000..8871307804d3f1e5c7cc49061614c69df26ab1ee --- /dev/null +++ b/data/fairseq/examples/hubert/simple_kmeans/dump_km_label.py @@ -0,0 +1,98 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import logging +import os +import sys + +import numpy as np + +import joblib +import torch +import tqdm + +logging.basicConfig( + format="%(asctime)s | %(levelname)s | %(name)s | %(message)s", + datefmt="%Y-%m-%d %H:%M:%S", + level=os.environ.get("LOGLEVEL", "INFO").upper(), + stream=sys.stdout, +) +logger = logging.getLogger("dump_km_label") + + +class ApplyKmeans(object): + def __init__(self, km_path): + self.km_model = joblib.load(km_path) + self.C_np = self.km_model.cluster_centers_.transpose() + self.Cnorm_np = (self.C_np ** 2).sum(0, keepdims=True) + + self.C = torch.from_numpy(self.C_np) + self.Cnorm = torch.from_numpy(self.Cnorm_np) + if torch.cuda.is_available(): + self.C = self.C.cuda() + self.Cnorm = self.Cnorm.cuda() + + def __call__(self, x): + if isinstance(x, torch.Tensor): + dist = ( + x.pow(2).sum(1, keepdim=True) + - 2 * torch.matmul(x, self.C) + + self.Cnorm + ) + return dist.argmin(dim=1).cpu().numpy() + else: + dist = ( + (x ** 2).sum(1, keepdims=True) + - 2 * np.matmul(x, self.C_np) + + self.Cnorm_np + ) + return np.argmin(dist, axis=1) + + +def get_feat_iterator(feat_dir, split, nshard, rank): + feat_path = f"{feat_dir}/{split}_{rank}_{nshard}.npy" + leng_path = f"{feat_dir}/{split}_{rank}_{nshard}.len" + with open(leng_path, "r") as f: + lengs = [int(line.rstrip()) for line in f] + offsets = [0] + np.cumsum(lengs[:-1]).tolist() + + def iterate(): + feat = np.load(feat_path, mmap_mode="r") + assert feat.shape[0] == (offsets[-1] + lengs[-1]) + for offset, leng in zip(offsets, lengs): + yield feat[offset: offset + leng] + + return iterate, len(lengs) + + +def dump_label(feat_dir, split, km_path, nshard, rank, lab_dir): + apply_kmeans = ApplyKmeans(km_path) + generator, num = get_feat_iterator(feat_dir, split, nshard, rank) + iterator = generator() + + lab_path = f"{lab_dir}/{split}_{rank}_{nshard}.km" + os.makedirs(lab_dir, exist_ok=True) + with open(lab_path, "w") as f: + for feat in tqdm.tqdm(iterator, total=num): + # feat = torch.from_numpy(feat).cuda() + lab = apply_kmeans(feat).tolist() + f.write(" ".join(map(str, lab)) + "\n") + logger.info("finished successfully") + + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser() + parser.add_argument("feat_dir") + parser.add_argument("split") + parser.add_argument("km_path") + parser.add_argument("nshard", type=int) + parser.add_argument("rank", type=int) + parser.add_argument("lab_dir") + args = parser.parse_args() + logging.info(str(args)) + + dump_label(**vars(args)) diff --git a/data/fairseq/examples/hubert/simple_kmeans/dump_mfcc_feature.py b/data/fairseq/examples/hubert/simple_kmeans/dump_mfcc_feature.py new file mode 100644 index 0000000000000000000000000000000000000000..c3537784d1d390701e96951d6e39f63f2023e32a --- /dev/null +++ b/data/fairseq/examples/hubert/simple_kmeans/dump_mfcc_feature.py @@ -0,0 +1,74 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import logging +import os +import sys + +import soundfile as sf +import torch +import torchaudio + +from feature_utils import get_path_iterator, dump_feature +from fairseq.data.audio.audio_utils import get_features_or_waveform + +logging.basicConfig( + format="%(asctime)s | %(levelname)s | %(name)s | %(message)s", + datefmt="%Y-%m-%d %H:%M:%S", + level=os.environ.get("LOGLEVEL", "INFO").upper(), + stream=sys.stdout, +) +logger = logging.getLogger("dump_mfcc_feature") + + +class MfccFeatureReader(object): + def __init__(self, sample_rate): + self.sample_rate = sample_rate + + def read_audio(self, path, ref_len=None): + wav = get_features_or_waveform(path, need_waveform=True, use_sample_rate=self.sample_rate) + if ref_len is not None and abs(ref_len - len(wav)) > 160: + logging.warning(f"ref {ref_len} != read {len(wav)} ({path})") + return wav + + def get_feats(self, path, ref_len=None): + x = self.read_audio(path, ref_len=ref_len) + with torch.no_grad(): + x = torch.from_numpy(x).float() + x = x.view(1, -1) + + mfccs = torchaudio.compliance.kaldi.mfcc( + waveform=x, + sample_frequency=self.sample_rate, + use_energy=False, + ) # (time, freq) + mfccs = mfccs.transpose(0, 1) # (freq, time) + deltas = torchaudio.functional.compute_deltas(mfccs) + ddeltas = torchaudio.functional.compute_deltas(deltas) + concat = torch.cat([mfccs, deltas, ddeltas], dim=0) + concat = concat.transpose(0, 1).contiguous() # (freq, time) + return concat + + +def main(tsv_dir, split, nshard, rank, feat_dir, sample_rate): + reader = MfccFeatureReader(sample_rate) + generator, num = get_path_iterator(f"{tsv_dir}/{split}.tsv", nshard, rank) + dump_feature(reader, generator, num, split, nshard, rank, feat_dir) + + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser() + parser.add_argument("tsv_dir") + parser.add_argument("split") + parser.add_argument("nshard", type=int) + parser.add_argument("rank", type=int) + parser.add_argument("feat_dir") + parser.add_argument("--sample_rate", type=int, default=16000) + args = parser.parse_args() + logger.info(args) + + main(**vars(args)) diff --git a/data/fairseq/examples/hubert/simple_kmeans/dump_w2v2_feature.py b/data/fairseq/examples/hubert/simple_kmeans/dump_w2v2_feature.py new file mode 100644 index 0000000000000000000000000000000000000000..a1f0d902acf0756580a1f4604feee8fc499a9a63 --- /dev/null +++ b/data/fairseq/examples/hubert/simple_kmeans/dump_w2v2_feature.py @@ -0,0 +1,95 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import logging +import os +import sys + +import fairseq +import soundfile as sf +import torch +import torch.nn.functional as F + +from feature_utils import get_path_iterator, dump_feature + + +logging.basicConfig( + format="%(asctime)s | %(levelname)s | %(name)s | %(message)s", + datefmt="%Y-%m-%d %H:%M:%S", + level=os.environ.get("LOGLEVEL", "INFO").upper(), + stream=sys.stdout, +) +logger = logging.getLogger("dump_w2v2_feature") + + +class Wav2Vec2FeatureReader(object): + def __init__(self, ckpt_path, layer, max_chunk=1600000): + ( + model, + cfg, + task, + ) = fairseq.checkpoint_utils.load_model_ensemble_and_task([ckpt_path]) + self.model = model[0].eval().cuda() + self.task = task + self.layer = layer # assume this is 1-based like HuBERT + self.max_chunk = max_chunk + logger.info(f"TASK CONFIG:\n{self.task.cfg}") + logger.info(f" max_chunk = {self.max_chunk}") + logger.info(f" model:\n{self.model}") + + def read_audio(self, path, ref_len=None): + wav, sr = sf.read(path) + assert sr == self.task.cfg.sample_rate, sr + if wav.ndim == 2: + wav = wav.mean(-1) + assert wav.ndim == 1, wav.ndim + if ref_len is not None and abs(ref_len - len(wav)) > 160: + logging.warning(f"ref {ref_len} != read {len(wav)} ({path})") + return wav + + def get_feats(self, path, ref_len=None): + x = self.read_audio(path, ref_len) + with torch.no_grad(): + x = torch.from_numpy(x).float().cuda() + if self.task.cfg.normalize: + x = F.layer_norm(x, x.shape) + x = x.view(1, -1) + + feat = [] + for start in range(0, x.size(1), self.max_chunk): + x_chunk = x[:, start: start + self.max_chunk] + res = self.model.extract_features( + source=x_chunk, + padding_mask=None, + mask=False, + layer=self.layer - 1, + ) + feat_chunk = res["x"] + feat.append(feat_chunk) + return torch.cat(feat, 1).squeeze(0) + + +def main(tsv_dir, split, ckpt_path, layer, nshard, rank, feat_dir, max_chunk): + reader = Wav2Vec2FeatureReader(ckpt_path, layer, max_chunk) + generator, num = get_path_iterator(f"{tsv_dir}/{split}.tsv", nshard, rank) + dump_feature(reader, generator, num, split, nshard, rank, feat_dir) + + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser() + parser.add_argument("tsv_dir") + parser.add_argument("split") + parser.add_argument("ckpt_path") + parser.add_argument("layer", type=int) + parser.add_argument("nshard", type=int) + parser.add_argument("rank", type=int) + parser.add_argument("feat_dir") + parser.add_argument("--max_chunk", type=int, default=1600000) + args = parser.parse_args() + logger.info(args) + + main(**vars(args)) diff --git a/data/fairseq/examples/hubert/simple_kmeans/feature_utils.py b/data/fairseq/examples/hubert/simple_kmeans/feature_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..f80bc4569768fac181133cdc8f76d1230e03bff6 --- /dev/null +++ b/data/fairseq/examples/hubert/simple_kmeans/feature_utils.py @@ -0,0 +1,66 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import logging +import os +import sys + +import tqdm +from npy_append_array import NpyAppendArray + + +logging.basicConfig( + format="%(asctime)s | %(levelname)s | %(name)s | %(message)s", + datefmt="%Y-%m-%d %H:%M:%S", + level=os.environ.get("LOGLEVEL", "INFO").upper(), + stream=sys.stdout, +) +logger = logging.getLogger("feature_utils") + + +def get_shard_range(tot, nshard, rank): + assert rank < nshard and rank >= 0, f"invaid rank/nshard {rank}/{nshard}" + start = round(tot / nshard * rank) + end = round(tot / nshard * (rank + 1)) + assert start < end, f"start={start}, end={end}" + logger.info( + f"rank {rank} of {nshard}, process {end-start} " + f"({start}-{end}) out of {tot}" + ) + return start, end + + +def get_path_iterator(tsv, nshard, rank): + with open(tsv, "r") as f: + root = f.readline().rstrip() + lines = [line.rstrip() for line in f] + start, end = get_shard_range(len(lines), nshard, rank) + lines = lines[start:end] + def iterate(): + for line in lines: + subpath, nsample = line.split("\t") + yield f"{root}/{subpath}", int(nsample) + return iterate, len(lines) + + +def dump_feature(reader, generator, num, split, nshard, rank, feat_dir): + iterator = generator() + + feat_path = f"{feat_dir}/{split}_{rank}_{nshard}.npy" + leng_path = f"{feat_dir}/{split}_{rank}_{nshard}.len" + + os.makedirs(feat_dir, exist_ok=True) + if os.path.exists(feat_path): + os.remove(feat_path) + + feat_f = NpyAppendArray(feat_path) + with open(leng_path, "w") as leng_f: + for path, nsample in tqdm.tqdm(iterator, total=num): + feat = reader.get_feats(path, nsample) + feat_f.append(feat.cpu().numpy()) + leng_f.write(f"{len(feat)}\n") + logger.info("finished successfully") + + diff --git a/data/fairseq/examples/hubert/simple_kmeans/learn_kmeans.py b/data/fairseq/examples/hubert/simple_kmeans/learn_kmeans.py new file mode 100644 index 0000000000000000000000000000000000000000..113ac655b8c0a585fe43797e99674e445098edd0 --- /dev/null +++ b/data/fairseq/examples/hubert/simple_kmeans/learn_kmeans.py @@ -0,0 +1,146 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import logging +import os +import sys + +import numpy as np +from sklearn.cluster import MiniBatchKMeans + +import joblib + +logging.basicConfig( + format="%(asctime)s | %(levelname)s | %(name)s | %(message)s", + datefmt="%Y-%m-%d %H:%M:%S", + level=os.environ.get("LOGLEVEL", "INFO").upper(), + stream=sys.stdout, +) +logger = logging.getLogger("learn_kmeans") + + +def get_km_model( + n_clusters, + init, + max_iter, + batch_size, + tol, + max_no_improvement, + n_init, + reassignment_ratio, +): + return MiniBatchKMeans( + n_clusters=n_clusters, + init=init, + max_iter=max_iter, + batch_size=batch_size, + verbose=1, + compute_labels=False, + tol=tol, + max_no_improvement=max_no_improvement, + init_size=None, + n_init=n_init, + reassignment_ratio=reassignment_ratio, + ) + + +def load_feature_shard(feat_dir, split, nshard, rank, percent): + feat_path = f"{feat_dir}/{split}_{rank}_{nshard}.npy" + leng_path = f"{feat_dir}/{split}_{rank}_{nshard}.len" + with open(leng_path, "r") as f: + lengs = [int(line.rstrip()) for line in f] + offsets = [0] + np.cumsum(lengs[:-1]).tolist() + + if percent < 0: + return np.load(feat_path, mmap_mode="r") + else: + nsample = int(np.ceil(len(lengs) * percent)) + indices = np.random.choice(len(lengs), nsample, replace=False) + feat = np.load(feat_path, mmap_mode="r") + sampled_feat = np.concatenate( + [feat[offsets[i]: offsets[i] + lengs[i]] for i in indices], axis=0 + ) + logger.info( + ( + f"sampled {nsample} utterances, {len(sampled_feat)} frames " + f"from shard {rank}/{nshard}" + ) + ) + return sampled_feat + + +def load_feature(feat_dir, split, nshard, seed, percent): + assert percent <= 1.0 + feat = np.concatenate( + [ + load_feature_shard(feat_dir, split, nshard, r, percent) + for r in range(nshard) + ], + axis=0, + ) + logging.info(f"loaded feature with dimension {feat.shape}") + return feat + + +def learn_kmeans( + feat_dir, + split, + nshard, + km_path, + n_clusters, + seed, + percent, + init, + max_iter, + batch_size, + tol, + n_init, + reassignment_ratio, + max_no_improvement, +): + np.random.seed(seed) + feat = load_feature(feat_dir, split, nshard, seed, percent) + km_model = get_km_model( + n_clusters, + init, + max_iter, + batch_size, + tol, + max_no_improvement, + n_init, + reassignment_ratio, + ) + km_model.fit(feat) + joblib.dump(km_model, km_path) + + inertia = -km_model.score(feat) / len(feat) + logger.info("total intertia: %.5f", inertia) + logger.info("finished successfully") + + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser() + parser.add_argument("feat_dir", type=str) + parser.add_argument("split", type=str) + parser.add_argument("nshard", type=int) + parser.add_argument("km_path", type=str) + parser.add_argument("n_clusters", type=int) + parser.add_argument("--seed", default=0, type=int) + parser.add_argument( + "--percent", default=-1, type=float, help="sample a subset; -1 for all" + ) + parser.add_argument("--init", default="k-means++") + parser.add_argument("--max_iter", default=100, type=int) + parser.add_argument("--batch_size", default=10000, type=int) + parser.add_argument("--tol", default=0.0, type=float) + parser.add_argument("--max_no_improvement", default=100, type=int) + parser.add_argument("--n_init", default=20, type=int) + parser.add_argument("--reassignment_ratio", default=0.0, type=float) + args = parser.parse_args() + logging.info(str(args)) + + learn_kmeans(**vars(args)) diff --git a/data/fairseq/examples/hubert/tests/sample.base.L9.km500.km b/data/fairseq/examples/hubert/tests/sample.base.L9.km500.km new file mode 100644 index 0000000000000000000000000000000000000000..656eef96e588b601a7a8c0f2ab8644d4185045fb --- /dev/null +++ b/data/fairseq/examples/hubert/tests/sample.base.L9.km500.km @@ -0,0 +1 @@ +17 17 17 17 296 296 20 20 20 461 461 20 184 20 20 20 184 289 144 445 445 213 213 213 213 252 215 129 401 20 354 180 494 44 416 416 416 192 192 180 180 84 84 84 16 88 88 88 88 319 242 240 348 35 35 117 404 197 226 209 83 55 55 55 322 67 94 199 118 118 118 118 118 118 402 219 219 219 222 222 222 353 59 245 245 251 251 241 241 431 367 367 178 35 35 35 458 192 351 41 324 324 324 252 464 464 139 139 424 424 424 497 497 497 122 90 42 42 147 380 380 499 319 319 319 348 348 33 33 394 90 76 465 74 425 425 386 386 431 319 319 319 319 319 240 203 53 473 34 340 340 340 340 116 64 212 384 377 123 123 123 216 216 216 114 114 57 57 57 203 381 381 117 48 13 47 80 20 80 80 320 7 7 364 345 141 141 141 141 281 281 9 86 221 198 198 22 283 455 236 239 239 107 107 395 286 286 286 468 468 406 406 467 176 176 176 328 200 200 248 464 145 365 365 365 365 330 385 457 77 77 77 54 224 300 334 334 382 304 304 271 186 31 342 342 342 198 22 283 5 38 162 232 232 482 68 26 26 359 359 81 444 213 213 252 143 458 41 324 324 324 422 143 445 445 445 351 180 486 315 315 450 450 450 203 53 473 291 89 116 379 243 478 478 66 482 482 105 105 336 336 354 29 498 498 498 498 396 396 313 37 314 198 22 222 222 222 222 245 129 74 74 437 437 496 496 496 413 94 199 41 41 324 324 318 318 269 342 9 168 106 106 284 426 426 426 426 348 64 76 401 259 108 123 153 153 153 153 372 372 396 313 24 314 90 401 259 445 445 351 351 365 365 365 365 282 282 215 233 233 229 427 20 247 126 126 126 326 326 326 326 326 326 326 101 101 101 149 228 228 20 289 20 7 217 70 65 189 189 151 240 285 300 300 495 406 467 176 135 135 339 248 466 114 222 222 222 313 313 239 384 371 490 490 38 31 54 54 224 494 494 236 129 259 74 190 487 288 288 288 288 374 173 173 280 280 302 302 175 175 69 69 223 130 129 401 75 108 119 295 295 295 295 143 192 192 135 135 135 135 200 200 464 255 255 255 251 251 241 431 235 235 235 348 348 465 192 44 44 236 8 8 354 319 319 383 348 36 310 107 107 395 462 462 8 32 32 32 354 153 153 153 153 153 387 387 387 387 85 207 318 318 318 49 453 9 168 125 125 125 125 125 466 199 44 44 143 129 144 445 351 351 351 486 486 460 285 285 302 302 497 497 122 239 161 161 79 79 499 499 499 265 265 265 85 85 85 299 299 173 352 352 427 229 170 247 15 15 15 15 15 15 193 193 193 17 diff --git a/data/fairseq/examples/hubert/tests/sample.base.L9.len b/data/fairseq/examples/hubert/tests/sample.base.L9.len new file mode 100644 index 0000000000000000000000000000000000000000..7d3028fa244a2121c51f39dcc92dc15be82823a6 --- /dev/null +++ b/data/fairseq/examples/hubert/tests/sample.base.L9.len @@ -0,0 +1 @@ +596 diff --git a/data/fairseq/examples/hubert/tests/sample.large.L20.len b/data/fairseq/examples/hubert/tests/sample.large.L20.len new file mode 100644 index 0000000000000000000000000000000000000000..7d3028fa244a2121c51f39dcc92dc15be82823a6 --- /dev/null +++ b/data/fairseq/examples/hubert/tests/sample.large.L20.len @@ -0,0 +1 @@ +596 diff --git a/data/fairseq/examples/hubert/tests/sample.large.hypo.word b/data/fairseq/examples/hubert/tests/sample.large.hypo.word new file mode 100644 index 0000000000000000000000000000000000000000..d77a4cfddcb93c2e08eb55e630c85fe840fd3cc2 --- /dev/null +++ b/data/fairseq/examples/hubert/tests/sample.large.hypo.word @@ -0,0 +1 @@ +KEEP A GOING AN IF YOU'RE LUCKY YOU'LL RUN PLUMB INTO THEM WAS THE JEERING ANSWER AS THE SLEEPY COWMEN SPURRED THEIR PONIES ON TOWARD CAMP MUTTERING THEIR DISAPPROVAL OF TAKING ALONG A BUNCH OF BOYS ON A CATTLE DRIVE (None-0) diff --git a/data/fairseq/examples/hubert/tests/sample.xlarge.L30.len b/data/fairseq/examples/hubert/tests/sample.xlarge.L30.len new file mode 100644 index 0000000000000000000000000000000000000000..7d3028fa244a2121c51f39dcc92dc15be82823a6 --- /dev/null +++ b/data/fairseq/examples/hubert/tests/sample.xlarge.L30.len @@ -0,0 +1 @@ +596 diff --git a/data/fairseq/examples/hubert/tests/sample.xlarge.hypo.word b/data/fairseq/examples/hubert/tests/sample.xlarge.hypo.word new file mode 100644 index 0000000000000000000000000000000000000000..53e402d4550c820220e0964654a600dabaca8b1c --- /dev/null +++ b/data/fairseq/examples/hubert/tests/sample.xlarge.hypo.word @@ -0,0 +1 @@ +KEEP A GOIN AND IF YOU'RE LUCKY YOU'LL RUN PLUMB INTO THEM WAS THE JEERING ANSWER AS THE SLEEPY COWMEN SPURRED THEIR PONIES ON TOWARD CAMP MUTTERING THEIR DISAPPROVAL OF TAKING ALONG A BUNCH OF BOYS ON A CATTLE DRIVE (None-0) diff --git a/data/fairseq/examples/hubert/tests/test_feature_and_unit.sh b/data/fairseq/examples/hubert/tests/test_feature_and_unit.sh new file mode 100644 index 0000000000000000000000000000000000000000..8cddb27758c97f0981e7c65696f0ef72fbb97471 --- /dev/null +++ b/data/fairseq/examples/hubert/tests/test_feature_and_unit.sh @@ -0,0 +1,92 @@ +#!/bin/bash + +set -e + +sizes="base large xlarge" + +declare -A ckpt_urls +ckpt_urls[base]="https://dl.fbaipublicfiles.com/hubert/hubert_base_ls960.pt" +ckpt_urls[large]="https://dl.fbaipublicfiles.com/hubert/hubert_large_ll60k.pt" +ckpt_urls[xlarge]="https://dl.fbaipublicfiles.com/hubert/hubert_xtralarge_ll60k.pt" + +declare -A km_layers +km_layers[base]=9 +km_layers[large]=20 +km_layers[xlarge]=30 + +declare -A km_urls +km_urls[base]="https://dl.fbaipublicfiles.com/hubert/hubert_base_ls960_L9_km500.bin" + +declare -A km_nunits +km_nunits[base]=500 + +test_dir=./examples/hubert/tests +split=sample + +echo -e "${test_dir}\n6313-76958-0021.flac\t190800" > "${test_dir}/${split}.tsv" + +check_feature () { + echo "checking features..." + + size=$1 + ckpt_url=$2 + km_layer=$3 + ckpt_path="$test_dir/$(basename "$ckpt_url")" + + if [ ! -f "$ckpt_path" ]; then + echo "downloading $ckpt_url to $ckpt_path" + wget "$ckpt_url" -O "$ckpt_path" + fi + + python ./examples/hubert/simple_kmeans/dump_hubert_feature.py \ + "${test_dir}" "${split}" "${ckpt_path}" "${km_layer}" 1 0 "${test_dir}" + + if diff -q "${test_dir}/${split}.${size}.L${km_layer}.npy" "${test_dir}/${split}_0_1.npy" &>/dev/null; then + echo "...passed npy check" + else + echo "...failed npy check" + fi + + if diff -q "${test_dir}/${split}.${size}.L${km_layer}.len" "${test_dir}/${split}_0_1.len" &>/dev/null; then + echo "...passed len check" + else + echo "...failed len check" + fi +} + + +check_unit () { + echo "checking units..." + + size=$1 + km_url=$2 + km_layer=$3 + km_nunit=$4 + km_path="$test_dir/$(basename "$km_url")" + + if [ ! -f "$km_path" ]; then + echo "downloading $km_url to $km_path" + wget "$km_url" -O "$km_path" + fi + + python ./examples/hubert/simple_kmeans/dump_km_label.py \ + "${test_dir}" "${split}" "${km_path}" 1 0 "${test_dir}" + + if diff -q "${test_dir}/${split}.${size}.L${km_layer}.km${km_nunit}.km" "${test_dir}/${split}_0_1.km" &>/dev/null; then + echo "...passed unit check" + else + echo "...failed unit check" + fi +} + + +for size in $sizes; do + echo "=== Running unit test for HuBERT $size ===" + check_feature "$size" "${ckpt_urls[$size]}" "${km_layers[$size]}" + + if [ -n "${km_urls[$size]}" ]; then + check_unit "$size" "${km_urls[$size]}" "${km_layers[$size]}" "${km_nunits[$size]}" + fi + + rm -f $test_dir/${split}_0_1.* +done diff --git a/data/fairseq/examples/hubert/tests/test_finetuned_asr.sh b/data/fairseq/examples/hubert/tests/test_finetuned_asr.sh new file mode 100644 index 0000000000000000000000000000000000000000..3c0538b1f734f8363d0aee6afa31a13e6bbb7306 --- /dev/null +++ b/data/fairseq/examples/hubert/tests/test_finetuned_asr.sh @@ -0,0 +1,46 @@ +#!/bin/bash + +set -e + +sizes="large xlarge" + +declare -A ckpt_urls +ckpt_urls[large]="https://dl.fbaipublicfiles.com/hubert/hubert_large_ll60k_finetune_ls960.pt" +ckpt_urls[xlarge]="https://dl.fbaipublicfiles.com/hubert/hubert_xtralarge_ll60k_finetune_ls960.pt" + +test_dir=$(pwd)/examples/hubert/tests +split=sample + +echo -e "${test_dir}\n6313-76958-0021.flac\t190800" > "${test_dir}/${split}.tsv" +echo -e "K E E P | A | G O I N G | A N D | I F | Y O U ' R E | L U C K Y | Y O U ' L L | R U N | P L U M B | I N T O | T H E M | W A S | T H E | J E E R I N G | A N S W E R | A S | T H E | S L E E P Y | C O W M E N | S P U R R E D | T H E I R | P O N I E S | O N | T O W A R D | C A M P | M U T T E R I N G | T H E I R | D I S A P P R O V A L | O F | T A K I N G | A L O N G | A | B U N C H | O F | B O Y S | O N | A | C A T T L E | D R I V E |" > "${test_dir}/${split}.ltr" + +check_asr () { + echo "checking asr outputs..." + + size=$1 + ckpt_url=$2 + ckpt_path="$test_dir/$(basename "$ckpt_url")" + + if [ ! -f "$ckpt_path" ]; then + echo "downloading $ckpt_url to $ckpt_path" + wget "$ckpt_url" -O "$ckpt_path" + fi + + python examples/speech_recognition/new/infer.py \ + --config-dir examples/hubert/config/decode --config-name infer_viterbi \ + common_eval.path="${ckpt_path}" task.data="${test_dir}" task.normalize=true \ + decoding.results_path="${test_dir}/pred" \ + common_eval.results_path="${test_dir}/pred" \ + common_eval.quiet=false dataset.gen_subset="${split}" + + if diff -q "${test_dir}/pred/hypo.word" "${test_dir}/${split}.${size}.hypo.word" &>/dev/null; then + echo "...passed word check" + else + echo "...failed word check" + fi + rm -rf "${test_dir}/pred" +} + +for size in $sizes; do + check_asr "$size" "${ckpt_urls[$size]}" +done diff --git a/data/fairseq/examples/hubert/update_ckpt.py b/data/fairseq/examples/hubert/update_ckpt.py new file mode 100644 index 0000000000000000000000000000000000000000..53c9e74ea613e30aa5c22614e658f2b7272bac0c --- /dev/null +++ b/data/fairseq/examples/hubert/update_ckpt.py @@ -0,0 +1,22 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import torch + +src_ckpt = "/checkpoint/wnhsu/w2v/archived/hubert_base_ls960_it2.pt" +ref_ckpt = "/checkpoint/wnhsu/w2v/hubert_icassp_oss_v3/iter2_km100-400k-grp-L6/oss.km500_p0_1_s334.pmw1_0.puw0_0.grpnorm.ml10.mp0_8.untie.mxsz250000.ufreq1.maxtok1400000.MU100k.s1337.ngpu32/checkpoint_last.pt" +new_ckpt = "/checkpoint/wnhsu/w2v/archived/hubert_base_ls960_it2_updated.pt" + + +def update_state(state): + state["model"]["label_embs_concat"] = state["model"].pop("label_embs") + state["args"].task = "hubert_pretraining" + state["args"].labels = f"['{state['args'].labels}']" + return state + + +src_state = torch.load(src_ckpt) +src_state = update_state(src_state) +torch.save(src_state, new_ckpt) diff --git a/data/fairseq/examples/language_model/README.adaptive_inputs.md b/data/fairseq/examples/language_model/README.adaptive_inputs.md new file mode 100644 index 0000000000000000000000000000000000000000..6650d58f37f320aa46402d59ce6494b2dd1c3faa --- /dev/null +++ b/data/fairseq/examples/language_model/README.adaptive_inputs.md @@ -0,0 +1,39 @@ +# Adaptive Input Representations for Neural Language Modeling (Baevski and Auli, 2018) + +## Pre-trained models + +Description | Parameters | Dataset | Model and Test set(s) +---|---:|---|--- +Adaptive Inputs
([Baevski and Auli, 2018](https://arxiv.org/abs/1809.10853)) | 1026M | [Google Billion Words](https://github.com/ciprian-chelba/1-billion-word-language-modeling-benchmark) | [download (.tar.bz2)](https://dl.fbaipublicfiles.com/fairseq/models/lm/adaptive_lm_gbw_huge.tar.bz2) +Adaptive Inputs
([Baevski and Auli, 2018](https://arxiv.org/abs/1809.10853)) | 247M | [WikiText-103](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/) | [download (.tar.bz2)](https://dl.fbaipublicfiles.com/fairseq/models/lm/adaptive_lm_wiki103.v2.tar.bz2) + +## Training an LM with adaptive inputs + +First, see the general [language modeling README](README.md) for instructions on +preprocessing the WikiText-103 data. + +Then use the following training command to train a model with adaptive inputs +using the `transformer_lm_wiki103` model architecture: +```bash +fairseq-train --task language_modeling \ + data-bin/wikitext-103 \ + --save-dir checkpoints/transformer_wikitext-103 \ + --arch transformer_lm_wiki103 \ + --max-update 286000 --lr 1.0 --t-mult 2 --lr-period-updates 270000 --lr-scheduler cosine --lr-shrink 0.75 \ + --warmup-updates 16000 --warmup-init-lr 1e-07 --stop-min-lr 1e-09 --optimizer nag --min-lr 0.0001 --clip-norm 0.1 \ + --criterion adaptive_loss --max-tokens 3072 --update-freq 3 --tokens-per-sample 3072 --seed 1 \ + --sample-break-mode none --skip-invalid-size-inputs-valid-test --ddp-backend=legacy_ddp +``` + +## Citation + +```bibtex +@inproceedings{ + baevski2018adaptive, + title={Adaptive Input Representations for Neural Language Modeling}, + author={Alexei Baevski and Michael Auli}, + booktitle={International Conference on Learning Representations}, + year={2019}, + url={https://openreview.net/forum?id=ByxZX20qFQ}, +} +``` diff --git a/data/fairseq/examples/language_model/README.conv.md b/data/fairseq/examples/language_model/README.conv.md new file mode 100644 index 0000000000000000000000000000000000000000..1ff8635906cf278208be4714e0ef805a6a6b4da1 --- /dev/null +++ b/data/fairseq/examples/language_model/README.conv.md @@ -0,0 +1,40 @@ +# Language Modeling with Gated Convolutional Networks (Dauphin et al., 2017) + +## Example usage + +First download and preprocess the data following the main [language modeling README](README.md). + +Then to train a convolutional LM using the `fconv_lm_dauphin_wikitext103` +architecture: +```bash +fairseq-train --task language_modeling \ + data-bin/wikitext-103 \ + --save-dir checkpoints/fconv_wikitext-103 \ + --arch fconv_lm_dauphin_wikitext103 \ + --adaptive-softmax-cutoff 10000,20000,200000 \ + --dropout 0.2 \ + --criterion adaptive_loss \ + --optimizer nag --clip-norm 0.1 --weight-decay 5e-06 \ + --lr 1.0 --lr-scheduler reduce_lr_on_plateau --lr-shrink 0.5 \ + --max-tokens 1024 --tokens-per-sample 1024 \ + --ddp-backend legacy_ddp \ + --max-epoch 35 +``` + +And evaluate with: +```bash +fairseq-eval-lm data-bin/wikitext-103 --path checkpoints/fconv_wiki103/checkpoint_best.pt +``` + +## Citation + +```bibtex +@inproceedings{dauphin2017language, + title={Language Modeling with Gated Convolutional Networks}, + author={Dauphin, Yann N and Fan, Angela and Auli, Michael and Grangier, David}, + booktitle={Proceedings of the 34th International Conference on Machine Learning-Volume 70}, + pages={933--941}, + year={2017}, + organization={JMLR} +} +``` diff --git a/data/fairseq/examples/language_model/README.md b/data/fairseq/examples/language_model/README.md new file mode 100644 index 0000000000000000000000000000000000000000..e78ea48e08dc99b69751923762107a8f8a9a5e3e --- /dev/null +++ b/data/fairseq/examples/language_model/README.md @@ -0,0 +1,123 @@ +# Neural Language Modeling + +## Pre-trained models + +Model | Description | Dataset | Download +---|---|---|--- +`transformer_lm.gbw.adaptive_huge` | Adaptive Inputs
([Baevski and Auli, 2018](https://arxiv.org/abs/1809.10853))
1026M params | [Google Billion Words](https://github.com/ciprian-chelba/1-billion-word-language-modeling-benchmark) | [download (.tar.bz2)](https://dl.fbaipublicfiles.com/fairseq/models/lm/adaptive_lm_gbw_huge.tar.bz2) +`transformer_lm.wiki103.adaptive` | Adaptive Inputs
([Baevski and Auli, 2018](https://arxiv.org/abs/1809.10853))
247M params | [WikiText-103](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset) | [download (.tar.bz2)](https://dl.fbaipublicfiles.com/fairseq/models/lm/adaptive_lm_wiki103.v2.tar.bz2) +`transformer_lm.wmt19.en` | English LM
([Ng et al., 2019](https://arxiv.org/abs/1907.06616)) | [WMT News Crawl](http://data.statmt.org/news-crawl/) | [download (.tar.gz)](https://dl.fbaipublicfiles.com/fairseq/models/lm/wmt19.en.tar.gz) +`transformer_lm.wmt19.de` | German LM
([Ng et al., 2019](https://arxiv.org/abs/1907.06616)) | [WMT News Crawl](http://data.statmt.org/news-crawl/) | [download (.tar.gz)](https://dl.fbaipublicfiles.com/fairseq/models/lm/wmt19.de.tar.gz) +`transformer_lm.wmt19.ru` | Russian LM
([Ng et al., 2019](https://arxiv.org/abs/1907.06616)) | [WMT News Crawl](http://data.statmt.org/news-crawl/) | [download (.tar.gz)](https://dl.fbaipublicfiles.com/fairseq/models/lm/wmt19.ru.tar.gz) + +## Example usage + +We require a few additional Python dependencies for preprocessing: +```bash +pip install fastBPE sacremoses +``` + +To sample from a language model using PyTorch Hub: +```python +import torch + +# List available models +torch.hub.list('pytorch/fairseq') # [..., 'transformer_lm.wmt19.en', ...] + +# Load an English LM trained on WMT'19 News Crawl data +en_lm = torch.hub.load('pytorch/fairseq', 'transformer_lm.wmt19.en', tokenizer='moses', bpe='fastbpe') +en_lm.eval() # disable dropout + +# Move model to GPU +en_lm.cuda() + +# Sample from the language model +en_lm.sample('Barack Obama', beam=1, sampling=True, sampling_topk=10, temperature=0.8) +# "Barack Obama is coming to Sydney and New Zealand (...)" + +# Compute perplexity for a sequence +en_lm.score('Barack Obama is coming to Sydney and New Zealand')['positional_scores'].mean().neg().exp() +# tensor(15.1474) + +# The same interface can be used with custom models as well +from fairseq.models.transformer_lm import TransformerLanguageModel +custom_lm = TransformerLanguageModel.from_pretrained('/path/to/model/dir', 'checkpoint100.pt', tokenizer='moses', bpe='fastbpe') +custom_lm.sample('Barack Obama', beam=5) +# "Barack Obama (...)" +``` + +## Training a transformer language model with the CLI tools + +### 1) Preprocess the data + +First download and prepare the [WikiText-103 dataset](https://www.salesforce.com/products/einstein/ai-research/the-wikitext-dependency-language-modeling-dataset/): +```bash +cd examples/language_model/ +bash prepare-wikitext-103.sh +cd ../.. +``` + +Next preprocess/binarize the data: +```bash +TEXT=examples/language_model/wikitext-103 +fairseq-preprocess \ + --only-source \ + --trainpref $TEXT/wiki.train.tokens \ + --validpref $TEXT/wiki.valid.tokens \ + --testpref $TEXT/wiki.test.tokens \ + --destdir data-bin/wikitext-103 \ + --workers 20 +``` + +### 2) Train a language model + +Next we'll train a basic transformer language model on wikitext-103. For more +advanced usage, see the [adaptive inputs README](README.adaptive_inputs.md). + +To train a basic LM (assumes 2 GPUs): +``` +$ fairseq-train --task language_modeling \ + data-bin/wikitext-103 \ + --save-dir checkpoints/transformer_wikitext-103 \ + --arch transformer_lm --share-decoder-input-output-embed \ + --dropout 0.1 \ + --optimizer adam --adam-betas '(0.9, 0.98)' --weight-decay 0.01 --clip-norm 0.0 \ + --lr 0.0005 --lr-scheduler inverse_sqrt --warmup-updates 4000 --warmup-init-lr 1e-07 \ + --tokens-per-sample 512 --sample-break-mode none \ + --max-tokens 2048 --update-freq 16 \ + --fp16 \ + --max-update 50000 +``` + +If you run out of memory, try reducing `--max-tokens` (max number of tokens per +batch) or `--tokens-per-sample` (max sequence length). You can also adjust +`--update-freq` to accumulate gradients and simulate training on a different +number of GPUs. + +### 3) Evaluate + +```bash +fairseq-eval-lm data-bin/wikitext-103 \ + --path checkpoints/transformer_wiki103/checkpoint_best.pt \ + --batch-size 2 \ + --tokens-per-sample 512 \ + --context-window 400 +# | Evaluated 245569 tokens in 56.1s (4379.02 tokens/s) +# | Loss: 3.4164, Perplexity: 30.46 +``` + +*Note:* The `--context-window` option controls how much context is provided to +each token when computing perplexity. When the window size is 0, the dataset is +chunked into segments of length 512 and perplexity is computed over each segment +normally. However, this results in worse (higher) perplexity since tokens that +appear earlier in each segment have less conditioning. When the maximum window +size is used (511 in this case), then we compute perplexity for each token +fully conditioned on 511 tokens of context. This slows down evaluation +significantly, since we must run a separate forward pass for every token in the +dataset, but results in better (lower) perplexity. + + +## Convolutional language models + +Please see the [convolutional LM README](README.conv.md) for instructions on +training convolutional language models. diff --git a/data/fairseq/examples/language_model/prepare-wikitext-103.sh b/data/fairseq/examples/language_model/prepare-wikitext-103.sh new file mode 100644 index 0000000000000000000000000000000000000000..751302156f0a6829af9c2ee5e0e2ca62c2cd4187 --- /dev/null +++ b/data/fairseq/examples/language_model/prepare-wikitext-103.sh @@ -0,0 +1,33 @@ +#!/bin/bash +# Adapted from https://github.com/facebookresearch/MIXER/blob/master/prepareData.sh + +URLS=( + "https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-v1.zip" +) +FILES=( + "wikitext-103-v1.zip" +) + +for ((i=0;i<${#URLS[@]};++i)); do + file=${FILES[i]} + if [ -f $file ]; then + echo "$file already exists, skipping download" + else + url=${URLS[i]} + wget "$url" + if [ -f $file ]; then + echo "$url successfully downloaded." + else + echo "$url not successfully downloaded." + exit -1 + fi + if [ ${file: -4} == ".tgz" ]; then + tar zxvf $file + elif [ ${file: -4} == ".tar" ]; then + tar xvf $file + elif [ ${file: -4} == ".zip" ]; then + unzip $file + fi + fi +done +cd .. diff --git a/data/fairseq/examples/m2m_100/README.md b/data/fairseq/examples/m2m_100/README.md new file mode 100644 index 0000000000000000000000000000000000000000..02a68a5f0919a26a0468069bed46a5b1abc78941 --- /dev/null +++ b/data/fairseq/examples/m2m_100/README.md @@ -0,0 +1,241 @@ +# Beyond English-Centric Multilingual Machine Translation + +## Introduction +In this work, we create a true Many-to-Many multilingual translation model that can translate directly between any pair of 100 languages. Our focus on non-English-Centric models brings gains of more than 10 BLEU when directly translating between non-English directions while performing competitively with the best single systems of WMT. + +If you are new to using fairseq, read the following walkthrough. Otherwise, skip to the sections below. + +0. **Generation Data** + +To download the generation data, follow the below commands. Note that all datasets need to be detokenized *before* applying SPM in the data preprocessing step. If you use these evaluation datasets, please cite their associated papers. +```bash +# WMT - use sacrebleu, example here: +sacrebleu -t wmt14 -l fr-en --echo src > wmt.test.fr-en.fr +sacrebleu -t wmt14 -l fr-en --echo ref > wmt.test.fr-en.en + +# WAT +wget http://lotus.kuee.kyoto-u.ac.jp/WAT/my-en-data/wat2020.my-en.zip +unzip wat2020.my-en.zip + +# FLORES +# download from: https://github.com/facebookresearch/flores + +# TED - need to detokenize with Moses! +# from: https://github.com/neulab/word-embeddings-for-nmt +wget http://phontron.com/data/ted_talks.tar.gz + +# Autshumato +# request to download: https://repo.sadilar.org/handle/20.500.12185/397 + +# Tatoeba Challenge +# available here: https://github.com/Helsinki-NLP/Tatoeba-Challenge +``` + +1. **Training Data** + +To produce the training data, we use a combination of [CCMatrix](https://arxiv.org/abs/1911.04944) and [CCAligned](https://arxiv.org/abs/1911.06154). Check out the instructions [here](https://github.com/facebookresearch/LASER/tree/master/tasks/CCMatrix) to download the raw data. + +2. **Preprocess Data** + +After downloading raw data, you will need to postprocess the data, then apply SPM, then binarize. Note that it is very important you run the postprocessing script, because this removes any instance of the evaluation data in the mined training data. + +```bash +# preprocess data + +# remove sentences with more than 50% punctuation +python /path/to/fairseq/examples/m2m_100/process_data/remove_too_much_punc.py + +# deduplicate training data +paste /path/to/datadir/train.$src /path/to/datadir/train.$tgt | awk '!x[$0]++' > /path/to/datadir/train.dedup +echo "keeping $(wc -l /path/to/datadir/train.dedup) bitext out of $(wc -l /path/to/datadir/train.$src)" +cut -f1 /path/to/datadir/train.dedup > /path/to/datadir/train.$src +cut -f2 /path/to/datadir/train.dedup > /path/to/datadir/train.$tgt + +# remove all instances of evaluation data from the training data +python /path/to/fairseq/examples/m2m_100/process_data/dedup_data.py + +# frequency cleaning +wget https://dl.fbaipublicfiles.com/m2m_100/histograms.tar.gz +tar -xvzf histograms.tar.gz +python /path/to/fairseq/examples/m2m_100/process_data/clean_histogram.py --src $src --tgt $tgt --src-file /path/to/source/file --tgt-file /path/to/output/file --src-output-file source_output.$src --tgt-output-file target_output.$tgt --histograms /path/to/histograms + +# apply SPM +wget https://dl.fbaipublicfiles.com/m2m_100/spm.128k.model +python /path/to/fairseq/scripts/spm_encode.py \ + --model spm.128k.model \ + --output_format=piece \ + --inputs=/path/to/input/file/here \ + --outputs=/path/to/output/file/here + +# length ratio cleaning +perl mosesdecoder/scripts/training/clean-corpus-n.perl --ratio 3 /path/to/training/data/train.spm.$src-$tgt $src $tgt /path/to/output/directory/train.spm.$src-$tgt 1 250 + +# binarize data +wget https://dl.fbaipublicfiles.com/m2m_100/data_dict.128k.txt +fairseq-preprocess \ + --source-lang $src --target-lang $tgt \ + --testpref spm.$src.$tgt \ + --thresholdsrc 0 --thresholdtgt 0 \ + --destdir data_bin \ + --srcdict data_dict.128k.txt --tgtdict data_dict.128k.txt +``` + +3. **Training Scripts** + +To reproduce the training of our models, we train with fairseq-py's multilingual translation [task](https://github.com/pytorch/fairseq/tree/main/examples/multilingual). If you are interested in model parallel training, also check out [fairscale](https://github.com/facebookresearch/fairscale). + +4. **Generation** + +To generate from our models, follow the the commands in the generation section below. + + +If you use any of the resources listed here, please cite: +```bibtex +@article{fan2020beyond, + title={Beyond English-Centric Multilingual Machine Translation}, + author={Fan, Angela and Bhosale, Shruti and Schwenk, Holger and Ma, Zhiyi and El-Kishky, Ahmed and Goyal, Siddharth and Baines, Mandeep and Celebi, Onur and Wenzek, Guillaume and Chaudhary, Vishrav and Goyal, Naman and Birch, Tom and Liptchinsky, Vitaliy and Edunov, Sergey and Grave, Edouard and Auli, Michael and Joulin, Armand}, + journal={arXiv preprint}, + year={2020} +} + +@article{schwenk2019ccmatrix, + title={Ccmatrix: Mining billions of high-quality parallel sentences on the web}, + author={Schwenk, Holger and Wenzek, Guillaume and Edunov, Sergey and Grave, Edouard and Joulin, Armand}, + journal={arXiv preprint arXiv:1911.04944}, + year={2019} +} + +@article{el2019massive, + title={A Massive Collection of Cross-Lingual Web-Document Pairs}, + author={El-Kishky, Ahmed and Chaudhary, Vishrav and Guzman, Francisco and Koehn, Philipp}, + journal={arXiv preprint arXiv:1911.06154}, + year={2019} +} +``` + + +## Trained Models + +### 418M and 1.2B Model +We include the last checkpoint for both of these models. + +```bash +wget https://dl.fbaipublicfiles.com/m2m_100/model_dict.128k.txt +wget https://dl.fbaipublicfiles.com/m2m_100/language_pairs_small_models.txt + +# 418M parameter model +wget https://dl.fbaipublicfiles.com/m2m_100/418M_last_checkpoint.pt + +# 1.2B parameter model +wget https://dl.fbaipublicfiles.com/m2m_100/1.2B_last_checkpoint.pt + +# Generation: +fairseq-generate $binarized_data_path --batch-size 32 --path $path_to_model --fixed-dictionary model_dict.128k.txt -s en -t fr --remove-bpe 'sentencepiece' --beam 5 --task translation_multi_simple_epoch --lang-pairs language_pairs_small_models.txt --decoder-langtok --encoder-langtok src --gen-subset test > gen_out +``` + +### 12B Model +12B parameter model trained on many-to-many training data for 100 languages. We include the last checkpoint, average of last 5 checkpoints, average of last 10 checkpoints. There isn't a universally best choice out of these three, but all three versions are pretty close in accuracy. You can either sweep over the 3 checkpoints on a dev test and use the best performing checkpoint for final testing. Or the last checkpoint can be a good default choice. + +**Model Download Links** +Configuration | 2 32GB GPUs | 4 16GB GPUs | 6 12GB GPUs | 8 8GB GPUs +:--|:--|:--|:--|:-- +Last Checkpoint | [12b_last_chk_2_gpus.pt](https://dl.fbaipublicfiles.com/m2m_100/12b_last_chk_2_gpus.pt) | [12b_last_chk_4_gpus.pt](https://dl.fbaipublicfiles.com/m2m_100/12b_last_chk_4_gpus.pt) | [12b_last_chk_6_gpus.pt](https://dl.fbaipublicfiles.com/m2m_100/12b_last_chk_6_gpus.pt) | [12b_last_chk_8_gpus.pt](https://dl.fbaipublicfiles.com/m2m_100/12b_last_chk_8_gpus.pt) +Average of last 5 checkpoints | [12b_avg5_chk_2_gpus.pt](https://dl.fbaipublicfiles.com/m2m_100/12b_avg5_chk_2_gpus.pt) | [12b_avg5_chk_4_gpus.pt](https://dl.fbaipublicfiles.com/m2m_100/12b_avg5_chk_4_gpus.pt) | [12b_avg5_chk_6_gpus.pt](https://dl.fbaipublicfiles.com/m2m_100/12b_avg5_chk_6_gpus.pt) | [12b_avg5_chk_8_gpus.pt](https://dl.fbaipublicfiles.com/m2m_100/12b_avg5_chk_8_gpus.pt) +Average of last 10 checkpoints | [12b_avg10_chk_2_gpus.pt](https://dl.fbaipublicfiles.com/m2m_100/12b_avg10_chk_2_gpus.pt) | [12b_avg10_chk_4_gpus.pt](https://dl.fbaipublicfiles.com/m2m_100/12b_avg10_chk_4_gpus.pt) | [12b_avg10_chk_6_gpus.pt](https://dl.fbaipublicfiles.com/m2m_100/12b_avg10_chk_6_gpus.pt) | [12b_avg10_chk_8_gpus.pt](https://dl.fbaipublicfiles.com/m2m_100/12b_avg10_chk_8_gpus.pt) + +**Generation Arguments** +Configuration | 2 32GB GPUs | 4 16GB GPUs | 6 12GB GPUs | 8 8GB GPUs +:--|:--|:--|:--|:-- +`--pipeline-encoder-balance` | `[26]` | `[1,15,10]` | `[1,9,9,7]` | `[1,6,6,6,7]` +`--pipeline-encoder-devices` | `[0]` | `[0,1,0]` | `[0,1,2,0]` | `[0,4,5,1,0]` +`--pipeline-decoder-balance` | `[3,22,1]` | `[3,11,11,1]` | `[3,7,7,8,1]` | `[1,6,6,6,6,1]` +`--pipeline-decoder-devices` | `[0,1,0]` | `[0,2,3,0]` | `[0,3,4,5,0]` | `[0,2,6,7,3,0]` + + +## SentencePiece Model + +```bash +wget https://dl.fbaipublicfiles.com/m2m_100/spm.128k.model +``` + +## Generation with M2M-100 + +### Encode using our SentencePiece Model + +Note: Install SentencePiece from [here](https://github.com/google/sentencepiece) + +```bash +fairseq=/path/to/fairseq +cd $fairseq +sacrebleu --echo src -l de-fr -t wmt19 | head -n 20 > raw_input.de-fr.de +sacrebleu --echo ref -l de-fr -t wmt19 | head -n 20 > raw_input.de-fr.fr +wget https://dl.fbaipublicfiles.com/m2m_100/spm.128k.model +for lang in de fr ; do + python scripts/spm_encode.py \ + --model spm.128k.model \ + --output_format=piece \ + --inputs=raw_input.de-fr.${lang} \ + --outputs=spm.de-fr.${lang} +done +``` + +### Binarization + +```bash +wget https://dl.fbaipublicfiles.com/m2m_100/data_dict.128k.txt +fairseq-preprocess \ + --source-lang de --target-lang fr \ + --testpref spm.de-fr \ + --thresholdsrc 0 --thresholdtgt 0 \ + --destdir data_bin \ + --srcdict data_dict.128k.txt --tgtdict data_dict.128k.txt +``` + +### Generation for the 12B model + +Note that generation can currently be run using 2 32GB / 4 16GB / 6 12GB / 8 8GB GPUs, and the corresponding model checkpoints and pipeline arguments can be found in the [12B Model Section](#12b-model). +Generation on CPUs will be added in the future. + +```bash +wget https://dl.fbaipublicfiles.com/m2m_100/model_dict.128k.txt +wget https://dl.fbaipublicfiles.com/m2m_100/language_pairs.txt +wget https://dl.fbaipublicfiles.com/m2m_100/12b_last_chk_4_gpus.pt +fairseq-generate \ + data_bin \ + --batch-size 1 \ + --path 12b_last_chk_4_gpus.pt \ + --fixed-dictionary model_dict.128k.txt \ + -s de -t fr \ + --remove-bpe 'sentencepiece' \ + --beam 5 \ + --task translation_multi_simple_epoch \ + --lang-pairs language_pairs.txt \ + --decoder-langtok --encoder-langtok src \ + --gen-subset test \ + --fp16 \ + --dataset-impl mmap \ + --distributed-world-size 1 --distributed-no-spawn \ + --pipeline-model-parallel \ + --pipeline-chunks 1 \ + --pipeline-encoder-balance '[1,15,10]' \ + --pipeline-encoder-devices '[0,1,0]' \ + --pipeline-decoder-balance '[3,11,11,1]' \ + --pipeline-decoder-devices '[0,2,3,0]' > gen_out +``` +## Evaluation with M2M-100 + +### Tokenization + +Note: Refer to tokenizers/README.md for more details on tokenization. + +```bash +cd ${fairseq}/examples/m2m_100 +cat ${fairseq}/gen_out | grep -P "^H" | sort -V | cut -f 3- | sh tok.sh fr > hyp +cat ${fairseq}/raw_input.de-fr.fr | sh tok.sh fr > ref +``` + +### BLEU + +```bash +sacrebleu -tok 'none' ref < hyp +``` diff --git a/data/fairseq/examples/m2m_100/install_dependecies.sh b/data/fairseq/examples/m2m_100/install_dependecies.sh new file mode 100644 index 0000000000000000000000000000000000000000..82a1054745264a56fbec4a8eb593884f8a42bd08 --- /dev/null +++ b/data/fairseq/examples/m2m_100/install_dependecies.sh @@ -0,0 +1,78 @@ +#!/usr/bin/env bash +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + + +CWD=`pwd` +INSTALL_PATH=$CWD/tokenizers/thirdparty + +MOSES=$INSTALL_PATH/mosesdecoder +if [ ! -d $MOSES ]; then + echo 'Cloning Moses github repository (for tokenization scripts)...' + git clone https://github.com/moses-smt/mosesdecoder.git $MOSES + cd $MOSES + # To deal with differences in handling ' vs " + git checkout 03578921cc1a03402 + cd - +fi + +WMT16_SCRIPTS=$INSTALL_PATH/wmt16-scripts +if [ ! -d $WMT16_SCRIPTS ]; then + echo 'Cloning Romanian tokenization scripts' + git clone https://github.com/rsennrich/wmt16-scripts.git $WMT16_SCRIPTS +fi + +KYTEA=$INSTALL_PATH/kytea +if [ ! -f $KYTEA/bin/kytea ]; then + git clone https://github.com/neubig/kytea.git $KYTEA + cd $KYTEA + autoreconf -i + ./configure --prefix=`pwd` + make + make install + cd .. +fi + +export MECAB=$INSTALL_PATH/mecab-0.996-ko-0.9.2 +if [ ! -f $MECAB/bin/mecab ]; then + cd $INSTALL_PATH + curl -LO https://bitbucket.org/eunjeon/mecab-ko/downloads/mecab-0.996-ko-0.9.2.tar.gz + tar zxfv mecab-0.996-ko-0.9.2.tar.gz + cd mecab-0.996-ko-0.9.2/ + ./configure --prefix=`pwd` + make + make install + + cd .. + curl -LO https://bitbucket.org/eunjeon/mecab-ko-dic/downloads/mecab-ko-dic-2.1.1-20180720.tar.gz + tar zxfv mecab-ko-dic-2.1.1-20180720.tar.gz + cd mecab-ko-dic-2.1.1-20180720/ + ./autogen.sh + ./configure --prefix=`pwd` --with-dicdir=$MECAB/lib/mecab/dic/mecab-ko-dic --with-mecab-config=$MECAB/bin/mecab-config + make + sh -c 'echo "dicdir=$MECAB/lib/mecab/dic/mecab-ko-dic" > $MECAB/etc/mecabrc' + make install + cd $CWD +fi + +INDIC_RESOURCES_PATH=$INSTALL_PATH/indic_nlp_resources +if [ ! -d $INDIC_RESOURCES_PATH ]; then + echo 'Cloning indic_nlp_resources' + git clone https://github.com/anoopkunchukuttan/indic_nlp_resources.git $INDIC_RESOURCES_PATH +fi + + +if [ ! -f $INSTALL_PATH/seg_my.py ]; then + cd $INSTALL_PATH + wget http://lotus.kuee.kyoto-u.ac.jp/WAT/my-en-data/wat2020.my-en.zip + unzip wat2020.my-en.zip + # switch to python3 + cat wat2020.my-en/myseg.py |sed 's/^sys.std/###sys.std/g' | sed 's/### sys/sys/g' | sed 's/unichr/chr/g' > seg_my.py + cd $CWD +fi + + +pip install pythainlp sacrebleu indic-nlp-library + diff --git a/data/fairseq/examples/m2m_100/process_data/clean_histogram.py b/data/fairseq/examples/m2m_100/process_data/clean_histogram.py new file mode 100644 index 0000000000000000000000000000000000000000..e24e073dc0eb43c76e2ce717f52bb848c5b026b8 --- /dev/null +++ b/data/fairseq/examples/m2m_100/process_data/clean_histogram.py @@ -0,0 +1,52 @@ +import argparse + +parser = argparse.ArgumentParser() +parser.add_argument('--src', type=str, help='Source language') +parser.add_argument('--tgt', type=str, help='Target language') +parser.add_argument('--src-file', type=str, help='Input source file') +parser.add_argument('--tgt-file', type=str, help='Input target file') +parser.add_argument('--src-output-file', type=str, help='Output source file') +parser.add_argument('--tgt-output-file', type=str, help='Output target file') +parser.add_argument('--threshold', type=float, default=0.5, help='Threshold') +parser.add_argument('--threshold-character', type=str, default=']', help='Threshold character') +parser.add_argument('--histograms', type=str, help='Path to histograms') + +args = parser.parse_args() + + +def read_hist(f): + ch = [] + for line in f: + c = line[0] + if c == args.threshold_character: + break + ch.append(c) + return ch + + +with(open("{}/{}".format(args.histograms, args.src), 'r', encoding='utf8')) as f: + ch1 = read_hist(f) + +with(open("{}/{}".format(args.histograms, args.tgt), 'r', encoding='utf8')) as f: + ch2 = read_hist(f) + +print("Accepted characters for {}: {}".format(args.src, ch1)) +print("Accepted characters for {}: {}".format(args.tgt, ch2)) + +with open(args.src_file, 'r', encoding='utf8') as fs1, open(args.tgt_file, 'r', encoding='utf8') as fs2, open(args.src_output_file, 'w', encoding='utf8') as fos1, open(args.tgt_output_file, 'w', encoding='utf8') as fos2: + ls1 = fs1.readline() + ls2 = fs2.readline() + + while ls1 or ls2: + cnt1 = len([c for c in ls1.strip() if c in ch1]) + cnt2 = len([c for c in ls2.strip() if c in ch2]) + + if cnt1 / len(ls1) > args.threshold and cnt2 / len(ls2) > args.threshold: + fos1.write(ls1) + fos2.write(ls2) + else: + print("{} {} {} \n{} {} {}".format(args.src, cnt1 / len(ls1), ls1.strip(), args.tgt, cnt2 / len(ls2), ls2.strip())) + + ls1 = fs1.readline() + ls2 = fs2.readline() + \ No newline at end of file diff --git a/data/fairseq/examples/m2m_100/process_data/dedup_data.py b/data/fairseq/examples/m2m_100/process_data/dedup_data.py new file mode 100644 index 0000000000000000000000000000000000000000..58d9ed1cd17b3ba70772a6d9adab709785495fd9 --- /dev/null +++ b/data/fairseq/examples/m2m_100/process_data/dedup_data.py @@ -0,0 +1,91 @@ +import argparse +from collections import namedtuple +import os + +DATADIR = "/path/to/train_data" +DEDUP_FROM_DIR = "/path/to/eval/data" +OUTPUT_DIR = "/path/to/output/data" + + +def main(args): + languages = set() + for language_directory in os.listdir(DATADIR): + if "_" in language_directory: + src, tgt = language_directory.split("_") + languages.add(LanguagePair(src=src, tgt=tgt)) + + data = existing_data() + train_languages = sorted(languages) + for language_pair in train_languages[args.start_index:args.start_index + args.size]: + print(language_pair) + dedup(language_pair, data) + + +LanguagePair = namedtuple("LanguagePair", ["src", "tgt"]) + + +def existing_data(): + data = set() + for file in os.listdir(DEDUP_FROM_DIR): + with open(os.path.join(DEDUP_FROM_DIR, file)) as f: + data |= set(f.readlines()) + return data + +def dedup(language_pair, data, verbose=True, output=True): + train_filenames = LanguagePair( + src=f"{DATADIR}/{language_pair.src}_{language_pair.tgt}/train.{language_pair.src}", + tgt=f"{DATADIR}/{language_pair.src}_{language_pair.tgt}/train.{language_pair.tgt}", + ) + + output_filenames = LanguagePair( + src=f"{OUTPUT_DIR}/train.dedup.{language_pair.src}-{language_pair.tgt}.{language_pair.src}", + tgt=f"{OUTPUT_DIR}/train.dedup.{language_pair.src}-{language_pair.tgt}.{language_pair.tgt}" + ) + + # If output exists, skip this pair. It has already been done. + if (os.path.exists(output_filenames.src) and + os.path.exists(output_filenames.tgt)): + if verbose: + print(f"{language_pair.src}-{language_pair.tgt} already done.") + return + + if verbose: + print(f"{language_pair.src}-{language_pair.tgt} ready, will check dups.") + + # If there is no output, no need to actually do the loop. + if not output: + return + + if os.path.exists(train_filenames.src) and os.path.exists(train_filenames.tgt): + with open(train_filenames.src) as f: + train_source = f.readlines() + + with open(train_filenames.tgt) as f: + train_target = f.readlines() + + # do dedup + new_train_source = [] + new_train_target = [] + for i, train_line in enumerate(train_source): + if train_line not in data and train_target[i] not in data: + new_train_source.append(train_line) + new_train_target.append(train_target[i]) + + assert len(train_source) == len(train_target) + assert len(new_train_source) == len(new_train_target) + assert len(new_train_source) <= len(train_source) + + with open(output_filenames.src, "w") as o: + for line in new_train_source: + o.write(line) + + with open(output_filenames.tgt, "w") as o: + for line in new_train_target: + o.write(line) + + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument("-s", "--start-index", required=True, type=int) + parser.add_argument("-n", "--size", required=True, type=int) + main(parser.parse_args()) diff --git a/data/fairseq/examples/m2m_100/process_data/remove_too_much_punc.py b/data/fairseq/examples/m2m_100/process_data/remove_too_much_punc.py new file mode 100644 index 0000000000000000000000000000000000000000..6c280de2403daffab477ac88e2008a68b9e61ff0 --- /dev/null +++ b/data/fairseq/examples/m2m_100/process_data/remove_too_much_punc.py @@ -0,0 +1,36 @@ +import gzip +import argparse +from string import punctuation + +def len_no_punc(s, punc): + return len([ch for ch in s if ch in punc]) + +def filter_overpunc(len_npunc, len_sen): + return len_npunc < 0.5*len_sen + +def main(args): + punc = punctuation + "—|–" + print('Processing file {}'.format(args.input)) + with gzip.open(args.input, 'rt', encoding=args.encoding) as tsv: + with open(args.bitext + '.' + args.src_lang, 'wt', encoding=args.encoding) as fsrc: + with open(args.bitext + '.' + args.tgt_lang, 'wt', encoding=args.encoding) as ftgt: + line = tsv.readline() + fields = line.split('\t') + + src, tgt = fields[1], fields[2] + + nchar_npunc_src = len_no_punc(src, punc) + nchar_npunc_tgt = len_no_punc(tgt, punc) + + if filter_overpunc(nchar_npunc_src, len(src)) and filter_overpunc(nchar_npunc_tgt, len(tgt)): + fsrc.write(src.strip() + '\n') + ftgt.write(tgt.strip() + '\n') + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument("--input", required=True, type=str) + parser.add_argument('--encoding', default='utf-8', help='character encoding for input/output') + parser.add_argument('--bitext', type=str, required=True, help='language direction') + parser.add_argument('--src-lang', type=str, required=True, help='Source language') + parser.add_argument('--tgt-lang', type=str, required=True, help='Target language') + main(parser.parse_args()) diff --git a/data/fairseq/examples/m2m_100/tok.sh b/data/fairseq/examples/m2m_100/tok.sh new file mode 100644 index 0000000000000000000000000000000000000000..ba2ec5a2f3f4794d2e528d3a6574bf05abe1d043 --- /dev/null +++ b/data/fairseq/examples/m2m_100/tok.sh @@ -0,0 +1,83 @@ +#!/usr/bin/env bash +# Copyright (c) 2019-present, Facebook, Inc. +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. +# + +set -e + +TOKENIZERS_SCRIPTS=tokenizers +INSTALL_PATH=$TOKENIZERS_SCRIPTS/thirdparty + +N_THREADS=8 + +lg=$1 + +MOSES=$INSTALL_PATH/mosesdecoder +REPLACE_UNICODE_PUNCT=$MOSES/scripts/tokenizer/replace-unicode-punctuation.perl +NORM_PUNC=$MOSES/scripts/tokenizer/normalize-punctuation.perl +REM_NON_PRINT_CHAR=$MOSES/scripts/tokenizer/remove-non-printing-char.perl +TOKENIZER=$MOSES/scripts/tokenizer/tokenizer.perl + +# special tokenization for Romanian +WMT16_SCRIPTS=$INSTALL_PATH/wmt16-scripts + +NORMALIZE_ROMANIAN=$WMT16_SCRIPTS/preprocess/normalise-romanian.py +REMOVE_DIACRITICS=$WMT16_SCRIPTS/preprocess/remove-diacritics.py + +# Burmese +MY_SEGMENT=$INSTALL_PATH/seg_my.py + +# Arabic +AR_TOKENIZER=$TOKENIZERS_SCRIPTS/tokenizer_ar.sh + +# Korean +KO_SEGMENT=$TOKENIZERS_SCRIPTS/seg_ko.sh + +# Japanese +JA_SEGMENT=$TOKENIZERS_SCRIPTS/seg_ja.sh + +# Indic +IN_TOKENIZER=$TOKENIZERS_SCRIPTS/tokenize_indic.py +INDIC_RESOURCES_PATH=$INSTALL_PATH/indic_nlp_resources + +# Thai +THAI_TOKENIZER=$TOKENIZERS_SCRIPTS/tokenize_thai.py + +# Chinese +CHINESE_TOKENIZER=$TOKENIZERS_SCRIPTS/tokenize_zh.py + +# Chinese +if [ "$lg" = "zh" ]; then + cat - | $REPLACE_UNICODE_PUNCT | $NORM_PUNC -l $lg | $REM_NON_PRINT_CHAR | python $CHINESE_TOKENIZER +# Thai +elif [ "$lg" = "th" ]; then + cat - | python $THAI_TOKENIZER +# Japanese +elif [ "$lg" = "ja" ]; then + cat - | $REPLACE_UNICODE_PUNCT | $NORM_PUNC -l $lg | $REM_NON_PRINT_CHAR | ${JA_SEGMENT} +# Korean +elif [ "$lg" = "ko" ]; then + cat - | $REM_NON_PRINT_CHAR | ${KO_SEGMENT} +# Romanian +elif [ "$lg" = "ro" ]; then + cat - | $REPLACE_UNICODE_PUNCT | $NORM_PUNC -l $lg | $REM_NON_PRINT_CHAR | $NORMALIZE_ROMANIAN | $REMOVE_DIACRITICS | $TOKENIZER -no-escape -threads $N_THREADS -l $lg +# Burmese +elif [ "$lg" = "my" ]; then + cat - | python ${MY_SEGMENT} +# Arabic +elif [ "$lg" = "ar" ]; then + cat - | ${AR_TOKENIZER} +# Indic +elif [ "$lg" = "ne" ]; then + cat - | python ${IN_TOKENIZER} $lg +elif [ "$lg" = "si" ]; then + cat - | python ${IN_TOKENIZER} $lg +elif [ "$lg" = "hi" ]; then + cat - | python ${IN_TOKENIZER} $lg +# other languages +else + cat - | $REPLACE_UNICODE_PUNCT | $NORM_PUNC -l $lg | $REM_NON_PRINT_CHAR | $TOKENIZER -no-escape -threads $N_THREADS -l $lg +fi diff --git a/data/fairseq/examples/m2m_100/tokenizers/README.md b/data/fairseq/examples/m2m_100/tokenizers/README.md new file mode 100644 index 0000000000000000000000000000000000000000..e116932bc80572f221cff6472a7b1eea7032925d --- /dev/null +++ b/data/fairseq/examples/m2m_100/tokenizers/README.md @@ -0,0 +1,18 @@ +# M2M-100 Tokenization + +We apply different tokenization strategies for different languages following the existing literature. Here we provide tok.sh a tokenizer that can be used to reproduce our results. + +To reproduce the results, follow these steps: + +``` +tgt_lang=... +reference_translation=... +cat generation_output | grep -P "^H" | sort -V | cut -f 3- | sh tok.sh $tgt_lang > hyp +cat $reference_translation |sh tok.sh $tgt_lang > ref +sacrebleu -tok 'none' ref < hyp +``` + +## Installation + +Tools needed for all the languages except Arabic can be installed by running install_dependencies.sh +If you want to evaluate Arabic models, please follow the instructions provided here: http://alt.qcri.org/tools/arabic-normalizer/ to install diff --git a/data/fairseq/examples/m2m_100/tokenizers/seg_ja.sh b/data/fairseq/examples/m2m_100/tokenizers/seg_ja.sh new file mode 100644 index 0000000000000000000000000000000000000000..be6f5ca5fe4ac8e8c786a439caaed1d1314f1aef --- /dev/null +++ b/data/fairseq/examples/m2m_100/tokenizers/seg_ja.sh @@ -0,0 +1,11 @@ +#!/usr/bin/env bash +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. +SCRIPT=`realpath $0` +KYTEA=`dirname $SCRIPT`/thirdparty/kytea +export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$KYTEA/lib:/usr/local/lib +export PATH=$PATH:"$KYTEA/bin" + +cat - | tr -d "[:blank:]" | kytea -notags diff --git a/data/fairseq/examples/m2m_100/tokenizers/seg_ko.sh b/data/fairseq/examples/m2m_100/tokenizers/seg_ko.sh new file mode 100644 index 0000000000000000000000000000000000000000..c523d92634d9b61b97bbcdbfd17dfc33465bfc09 --- /dev/null +++ b/data/fairseq/examples/m2m_100/tokenizers/seg_ko.sh @@ -0,0 +1,12 @@ +#!/usr/bin/env bash +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. +SCRIPT=`realpath $0` +MECAB=`dirname $SCRIPT`/thirdparty/mecab-0.996-ko-0.9.2 + +export PATH=$PATH:"$MECAB/bin":"$MECAB/lib" +export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:"$MECAB/lib" + +cat - | mecab -O wakati diff --git a/data/fairseq/examples/m2m_100/tokenizers/thirdparty/.gitignore b/data/fairseq/examples/m2m_100/tokenizers/thirdparty/.gitignore new file mode 100644 index 0000000000000000000000000000000000000000..19eb6a9dd705ac583f22ecb60d9b744987e27ff1 --- /dev/null +++ b/data/fairseq/examples/m2m_100/tokenizers/thirdparty/.gitignore @@ -0,0 +1,12 @@ +seg_my.py +indic_nlp_library/ +indic_nlp_resources/ +kytea/ +mecab-0.996-ko-0.9.2.tar.gz +mecab-0.996-ko-0.9.2/ +mosesdecoder/ +wat2020.my-en.zip +wat2020.my-en/ +wmt16-scripts/ +mecab-ko-dic-2.1.1-20180720/ +mecab-ko-dic-2.1.1-20180720.tar.gz \ No newline at end of file diff --git a/data/fairseq/examples/m2m_100/tokenizers/tokenize_indic.py b/data/fairseq/examples/m2m_100/tokenizers/tokenize_indic.py new file mode 100644 index 0000000000000000000000000000000000000000..a44fad07f7c718f99cccd445f33c62b0e3c562f4 --- /dev/null +++ b/data/fairseq/examples/m2m_100/tokenizers/tokenize_indic.py @@ -0,0 +1,23 @@ +#!/usr/bin/env python3 +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +# Use: echo {text} | python tokenize_indic.py {language} + +import sys + +from indicnlp.normalize.indic_normalize import IndicNormalizerFactory +from indicnlp.tokenize.indic_tokenize import trivial_tokenize + + +factory = IndicNormalizerFactory() +normalizer = factory.get_normalizer( + sys.argv[1], remove_nuktas=False, nasals_mode="do_nothing" +) + +for line in sys.stdin: + normalized_line = normalizer.normalize(line.strip()) + tokenized_line = " ".join(trivial_tokenize(normalized_line, sys.argv[1])) + print(tokenized_line) diff --git a/data/fairseq/examples/m2m_100/tokenizers/tokenize_thai.py b/data/fairseq/examples/m2m_100/tokenizers/tokenize_thai.py new file mode 100644 index 0000000000000000000000000000000000000000..9c72cb89056f6fc92a8963415e5f3a1e61b33a5b --- /dev/null +++ b/data/fairseq/examples/m2m_100/tokenizers/tokenize_thai.py @@ -0,0 +1,13 @@ +#!/usr/bin/env python3 +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import sys + +from pythainlp import word_tokenize + + +for line in sys.stdin: + print(" ".join(word_tokenize(line.strip()))) diff --git a/data/fairseq/examples/m2m_100/tokenizers/tokenize_zh.py b/data/fairseq/examples/m2m_100/tokenizers/tokenize_zh.py new file mode 100644 index 0000000000000000000000000000000000000000..674b5849cba829cf4f07a69369e9cc6eed376d4c --- /dev/null +++ b/data/fairseq/examples/m2m_100/tokenizers/tokenize_zh.py @@ -0,0 +1,14 @@ +#!/usr/bin/env python3 +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + + +import fileinput + +import sacrebleu + + +for line in fileinput.input(): + print(sacrebleu.tokenize_zh(line)) diff --git a/data/fairseq/examples/m2m_100/tokenizers/tokenizer_ar.sh b/data/fairseq/examples/m2m_100/tokenizers/tokenizer_ar.sh new file mode 100644 index 0000000000000000000000000000000000000000..ad35d7adf28dc9b23d13a6a3fec0b12cb760e855 --- /dev/null +++ b/data/fairseq/examples/m2m_100/tokenizers/tokenizer_ar.sh @@ -0,0 +1,27 @@ +#!/usr/bin/env sh +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. +# +# Please follow the instructions here http://alt.qcri.org/tools/arabic-normalizer/ +# to install tools needed for Arabic + +echo "Please install Arabic tools: http://alt.qcri.org/tools/arabic-normalizer/" +echo "Then update environment variables in tokenizer_ar.sh" +exit 1 + +SVMTOOL=... +GOMOSESGO=... +QCRI_ARABIC_NORMALIZER=... + +export PERL5LIB="$SVMTOOL/lib":"$GOMOSESGO/bin/MADA-3.2":$PERL5LIB + + +tempfile=$(mktemp) +cat - > $tempfile + +cd $QCRI_ARABIC_NORMALIZER + +bash qcri_normalizer_mada3.2_aramorph1.2.1.sh $tempfile +cat $tempfile.mada_norm-aramorph.europarl_tok diff --git a/data/fairseq/examples/moe_lm/README.md b/data/fairseq/examples/moe_lm/README.md new file mode 100644 index 0000000000000000000000000000000000000000..64ee57ec22092cfd2455622f4daf7681fd034be3 --- /dev/null +++ b/data/fairseq/examples/moe_lm/README.md @@ -0,0 +1,126 @@ +# Efficient Large Scale Language Modeling with Mixtures of Experts + +## Introduction + +Mixture of Experts layers (MoEs) enable efficient scaling of language models +through conditional computation. This work empirically compares how +autoregressive MoE language models scale in comparison with dense models in a +wide range of settings: in- and out-of-domain language modeling, zero- and +few-shot priming, and full fine-tuning. See the associated paper for more +details. + +This repo contains instructions for reproducing results from the paper. + +## Pre-trained models + +These models are intended for research purposes only in order to reproduce the +results from the paper, and to enable further research on the capabilities and +limitations of language models. Please see the [model card](model_card.md) for +more details about how the models were trained and evaluated, as well as their +limitations and intended use. + +#### Dense models + +Dense models can be run directly from the `main` branch. + +Model | Layers | Model Dim | Languages | Download +---|---|---|---|--- +`dense_125m` | 12 | 768 | English | [en_dense_lm_125m.tar.gz (0.2GB)](https://dl.fbaipublicfiles.com/fairseq/models/lm/en_dense_lm_125m.tar.gz) +`dense_355m` | 24 | 1024 | English | [en_dense_lm_355m.tar.gz (0.6GB)](https://dl.fbaipublicfiles.com/fairseq/models/lm/en_dense_lm_355m.tar.gz) +`dense_1_3b` | 24 | 2048 | English | [en_dense_lm_1_3b.tar.gz (2.3GB)](https://dl.fbaipublicfiles.com/fairseq/models/lm/en_dense_lm_1_3b.tar.gz) +`dense_2_7b` | 32 | 2560 | English | [en_dense_lm_2_7b.tar.gz (4.6GB)](https://dl.fbaipublicfiles.com/fairseq/models/lm/en_dense_lm_2_7b.tar.gz) +`dense_6_7b` | 32 | 4096 | English | [en_dense_lm_6_7b.tar.gz (12GB)](https://dl.fbaipublicfiles.com/fairseq/models/lm/en_dense_lm_6_7b.tar.gz) +`dense_13b` | 40 | 5120 | English | [en_dense_lm_13b.tar.gz (23GB)](https://dl.fbaipublicfiles.com/fairseq/models/lm/en_dense_lm_13b.tar.gz) + +#### Mixture of expert models + +MoE models must be run from the `moe` branch. Please see the +[MoE README](https://github.com/pytorch/fairseq/tree/moe#evaluating-moe-language-models) +for more details about how to load and evaluate MoE models. + +Model | Layers | Model Dim | Languages | Download +---|---|---|---|--- +`moe_15b` | 12 | 768 | English | [en_moe_lm_15b.tar.gz](https://dl.fbaipublicfiles.com/fairseq/models/lm/en_moe_lm_15b.tar.gz) +`moe_52b` | 24 | 1024 | English | [en_moe_lm_52b.tar.gz](https://dl.fbaipublicfiles.com/fairseq/models/lm/en_moe_lm_52b.tar.gz) +`moe_207b` | 24 | 2048 | English | Available by request +`moe_1_1t` | 32 | 4096 | English | Available by request + +## Evaluation + +### Example (COPA) + +The following snippet shows how to evaluate our dense models on the [Choice of +Plausible Alternatives (COPA)](https://people.ict.usc.edu/~gordon/copa.html) task. + +```python +from fairseq.models.transformer_lm import TransformerLanguageModel +model_dir = '/path/to/en_dense_lm_125m' +lm = TransformerLanguageModel.from_pretrained(model_dir, bpe='gpt2') +lm = lm.eval(); # disable dropout +lm = lm.half(); # use FP16 for evaluation +lm = lm.cuda(); # move to GPU + +def get_logprobs(prompt): + import re + prompt = re.sub('\n+' , '\n', prompt) # collapse repeated newlines, which indicate separate documents + return lm.score(prompt, replace_newlines_with_eos=True)['positional_scores'] + +# Zero-shot evaluation for the Choice of Plausible Alternatives (COPA) task. +# A return value of 1 indicates that the first alternative is more plausible, +# while 2 indicates that the second alternative is more plausible. +def COPA_eval(prompt, alternative1, alternative2): + lprob1 = get_logprobs(prompt + "\n" + alternative1).sum() + lprob2 = get_logprobs(prompt + "\n" + alternative2).sum() + return 1 if lprob1 > lprob2 else 2 + +COPA_eval("The man broke his toe. What was the CAUSE of this?", "He got a hole in his sock.", "He dropped a hammer on his foot.") +# 2 +COPA_eval("I tipped the bottle. What happened as a RESULT?", "The liquid in the bottle froze.", "The liquid in the bottle poured out.") +# 2 +COPA_eval("I knocked on my neighbor's door. What happened as a RESULT?", "My neighbor invited me in.", "My neighbor left his house.") +# 1 +``` + +### Data format + +Few-shot prompting is known to be sensitive to the input formatting, and it is usually best to match the formatting used in pretraining. + +During pretraining our models were presented with data in the following format (i.e., one paragraph per line, with a blank line separating documents): +``` + ... + ... + + ... +... +``` + +#### Newlines + +While we use the byte-level BPE from GPT-2/3, fairseq's preprocessing replaces newlines with the end-of-sentence symbol (``), which corresponds to embedding index `2`. +Thus **the model never saw newline characters during pretraining** and newlines should not be used during few-shot prompting. + +This is more clearly illustrated in the following example, which uses fairseq's Hub Interface to tokenize two documents in the desired format: +```python +from fairseq.models.transformer_lm import TransformerLanguageModel +model_dir = '/path/to/en_dense_lm_125m' +lm = TransformerLanguageModel.from_pretrained(model_dir, bpe='gpt2') + +data = """\ +This is the first paragraph of the first document. +This is the second paragraph of the first document. + +This is the first paragraph of the second document.\ +""" + +# The following is wrong, since it will encode newlines present in `data`. +tokens_bad = lm.score(data)['tokens'] +assert '\n' in lm.decode(tokens_bad) # oops, we encoded a newline + +# Instead pass the replace_newlines_with_eos option to get the correct behavior. +tokens_good = lm.score(data, replace_newline_with_eos=True)['tokens'] +assert '\n' not in lm.decode(tokens_good) # no newlines were encoded +``` + +## Citation + +Coming soon. diff --git a/data/fairseq/examples/moe_lm/data_card.md b/data/fairseq/examples/moe_lm/data_card.md new file mode 100644 index 0000000000000000000000000000000000000000..54e694b62088f467483504f727a23565b894e0a2 --- /dev/null +++ b/data/fairseq/examples/moe_lm/data_card.md @@ -0,0 +1,221 @@ +# Data card for the paper "Efficient Large Scale Language Modeling with Mixtures of Experts" +## Version 1.0.0 + +We follow the recommendations of Gebru et al. (2018) and provide a datacard for the dataset used to train the 1.1T parameter model. + +## Motivation +* **For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description.** +The pre-training data for training the 1.1 T model was created by a union of six English language datasets, including five datasets used by RoBERTa (Liu et al 2019) and the English subset of CC 100. These purpose of creating this dataset was to pre-train the language model. + +* **Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?** +FAIR (Fundamental Artificial Intelligence Research) + +* **Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number.** +FAIR (Fundamental Artificial Intelligence Research) + +* **Any other comments?** +No. + +## Composition + +* **What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.** +The instances are textual documents. The overall dataset is composed from a union of the following datasets - + * BookCorpus (Zhu et al., 2019) consists of more than 10K unpublished books (4GB); + * English Wikipedia, excluding lists, tables and headers (12GB); + * CC-News (Nagel,2016) contains 63 million English news articles crawled between September 2016 and February 2019 (76GB); + * OpenWebText (Gokaslan and Cohen, 2019), an open source recreation of the WebText dataset used to train GPT-2 (38GB); + * CC-Stories (Trinh and Le, 2018) contains a subset of CommonCrawl data filtered to match the story-like style of Winograd schemas (31GB); + * English CC100 (Wenzek et al., 2020), a dataset extracted from CommonCrawl snapshots between January 2018 and December 2018, filtered to match the style of Wikipedia (292GB). + +* **How many instances are there in total (of each type, if appropriate)?** +The training data contains 112B tokens corresponding to 453 GB of data. + +* **Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).** +The English CC100 section of the dataset is a subset of CommonCrawl snapshots extracted between January 2018 to December 2018, filtered to match the style of Wikipedia. The CC-stories dataset contains a subset of CommonCrawl data filtered to match the story-like style of Winograd schemas. + +* **What data does each instance consist of? “Raw” data (e.g., unprocessed text or images) or features? In either case, please provide a description.** +Each instance consists of raw text data. + +* **Is there a label or target associated with each instance? If so, please provide a description.** +No. + +* **Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text.** +No. + +* **Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? If so, please describe how these relationships are made explicit.** +There are no explicit relationships between individual instances. + +* **Are there recommended data splits (e.g., training, development/validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them.** +We hold out a random validation set of approximately 150MB from the pretraining data, sampled proportionally to each dataset's size in the pretraining corpus. + +* **Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description.** +N/A + +* **Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)?** +It's self-contained. + +* **Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)? If so, please provide a description.** +The datasets used are publicly available, and the information in them is not considered confidential. + +* **Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why.** +Parts of the dataset are a subset of public Common Crawl data, which could contain sentences that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety. + +* **Does the dataset relate to people? If not, you may skip the remaining questions in this section.** +Some documents of this data relate to people, such as news articles, Wikipedia descriptions, etc. + +* **Does the dataset identify any subpopulations (e.g., by age, gender)? If so, please describe how these subpopulations are identified and provide a description of their respective distributions within the dataset.** +No. + +* **Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? If so, please describe how** +In addition to individuals who have Wikipedia pages (celebrities, politicians, etc.), it may be possible to identify other individuals by their names, Twitter account names, etc. if that information is present in Common Crawl. + +* **Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals racial or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide a description.** +The training dataset is partially derived from Common Crawl, which may contain some sensitive information. + +* **Any other comments?** +No + + +## Collection Process + +* **How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/ derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how.** +N/A. The dataset is a union of six publicly available datasets. + +* **What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated?** +N/A + +* **If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?** +Please refer to the main document for details. + +* **Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?** +This data is mined, filtered and sampled by machines. + +* **Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.** +Different parts of the dataset were mined over different time periods. +1. The CC-News dataset contains English news articles crawled between September 2016 and February 2019. +2. The English CC-100 dataset was extracted from CommonCrawl snapshots between January 2018 and December 2018. + +* **Were any ethical review processes conducted (e.g., by an institutional review board)? If so, please provide a description of these review processes, including the outcomes, as well as a link or other access point to any supporting documentation.** +No. + +* **Does the dataset relate to people? If not, you may skip the remainder of the questions in this section.** +No. + +* **Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)?** +N/A + +* **Were the individuals in question notified about the data collection? If so, please describe (or show with screenshots or other information) how notice was provided, and provide a link or other access point to, or otherwise reproduce, the exact language of the notification itself.** +N/A + +* **Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide a link or other access point to, or otherwise reproduce, the exact language to which the individuals consented.** +N/A + +* **If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? If so, please provide a description, as well as a link or other access point to the mechanism (if appropriate).** +N/A + +* **Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? If so, please provide a description of this analysis, including the outcomes, as well as a link or other access point to any supporting documentation.** +Some responsible AI related evaluations were performed. Please refer to the main document and the model card for the paper. + +* **Any other comments?** +No + + +## Preprocessing/cleaning/labeling + + +* **Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remainder of the questions in this section.** +The component datasets went through standard cleaning and re-formatting practices, including removing repetitive/non informative text like "Chapter One", or "This ebook by Project Gutenberg". + +* **Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the “raw” data.** +The "raw" component datasets is publicly available in their respective locations (more details can be seen in the respective papers linked in references). + +* **Is the software used to preprocess/clean/label the instances available? If so, please provide a link or other access point.** +The software is proprietary to Meta Platforms and currently unavailable publicly. + +* **Any other comments?** +No + + +## Uses + +* **Has the dataset been used for any tasks already? If so, please provide a description.** +Yes, this dataset was used to pre-train the models described in the paper. + +* **Is there a repository that links to any or all papers or systems that use the dataset? If so, please provide a link or other access point.** +No. + +* **What (other) tasks could the dataset be used for?** +This data can be used to pretrain English language models, which are foundation to many current and future language tasks. + +* **Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? For example, is there anything that a future user might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other undesirable harms (e.g., financial harms, legal risks) If so, please provide a description. Is there anything a future user could do to mitigate these undesirable harms?** +The pipeline for creating this dataset paves a way for building a scalable infrastructure for mining datasets to be be used for training large-scale models. + +* **Are there tasks for which the dataset should not be used? If so, please provide a description.** +No. + +* **Any other comments?** +No. + +## Distribution + + +* **Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description.** +No. + +* **How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)?** +N/A + +* **When will the dataset be distributed?** +No. + +* **Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.** +No. + +* **Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions.** +No. + +* **Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation.** +N/A + +* **Any other comments?** +No. + +## Maintenance + +* **Who is supporting/hosting/maintaining the dataset?** +FAIR (Fundamental Artificial Intelligence Research) + +* **How can the owner/curator/manager of the dataset be contacted (e.g., email address)?** +Refer to the main document. + +* **Is there an erratum? If so, please provide a link or other access point.** +N/A + +* **Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to users (e.g., mailing list, GitHub)?** +No plan for updating. + +* **If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced.** +N/A + +* **Will older versions of the dataset continue to be supported/hosted/maintained? If so, please describe how. If not, please describe how its obsolescence will be communicated to users.** +N/A + +* **If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? If so, please provide a description. Will these contributions be validated/ verified? If so, please describe how. If not, why not? Is there a process for communicating/ distributing these contributions to other users? If so, please provide a description.** +No. + +* **Any other comments?** +No. + +## References +Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. + +Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2019. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. arXiv:1506.06724. + +Sebastian Nagel. 2016. Cc-news. http: //web.archive.org/save/http: //commoncrawl.org/2016/10/news-dataset-available. + +Aaron Gokaslan and Vanya Cohen. 2019. Openwebtext corpus. http://web.archive.org/save/http://Skylion007.github.io/OpenWebTextCorpus + +Trieu H Trinh and Quoc V Le. 2018. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847. + +Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4003–4012, Marseille, France. European Language Resources Association. + diff --git a/data/fairseq/examples/moe_lm/model_card.md b/data/fairseq/examples/moe_lm/model_card.md new file mode 100644 index 0000000000000000000000000000000000000000..a1cd68116aeff8d401ddd96461b7159151840ac9 --- /dev/null +++ b/data/fairseq/examples/moe_lm/model_card.md @@ -0,0 +1,170 @@ +# Model card for the paper ``Efficient Large Scale Language Modeling with Mixtures of Experts" +## Version 1.0.0 + +### Model developer +FAIR (Fundamental Artificial Intelligence Research) + +### Model type +An autoregressive English language model trained on a union of six English language models. We explore dense and sparse (MoE based) architectures in the paper. +* Dense models - Our dense models range from 125M parameters to 13B parameters. +* Sparse (MoE) models - Our MoE based models range from 15B parameters to 1.1 Trillion parameters. +This model card focuses on the 1.1 Trillion parameter model, but the discussion +applies to all of the models explored in this work. + +### Citation details +Artetxe et al. (2021): Efficient Large Scale Language Modeling with Mixtures of Experts + +### Model Feedback Channel +fairseq + +## Intended use +### Primary intended use +For research purposes only, e.g. reproducing model evaluation results. Generation is only used in a limited capacity for explanation/justification or for prompting/probing/priming for class labels. + +### Out of scope uses +The primary purpose of the model is not to generate language, although the model is capable of doing that. + +## Factors influencing model performance +This section discusses potential risks associated with using the model. + +### Relevant factors +Based on known problems with NLP technology, potential relevant factors include bias (gender, profession, race and religion). + +### Evaluation factors +The 1.1T model was evaluated on StereoSet and CrowS-Pairs datasets to quantify encoded bias in the model. + +## Metrics +### Model performance measures +The 1.1T parameter model was primarily evaluated on +1. In-domain and out-of-domain language modeling perplexity. +2. Zero-shot and few-shot priming. +3. Fully supervised finetuning. + +### Approaches to handle uncertainty +For few-shot learning, we report the average results across 25 runs, randomly sampling a different set of few-shot examples from the training set each time. + +## Evaluation data +## Zero Shot evaluation + +### HellaSwag +#### Description +HellaSwag is a dataset for evaluating commonsense reasoning. + +### PIQA +#### Description +PIQA is a dataset designed to evaluate reasoning about Physical Commonsense in Natural Language + +### ReCoRd +#### Description +Reading Comprehension with Commonsense Reasoning Dataset (ReCoRD) is a large-scale reading comprehension dataset which requires commonsense reasoning. ReCoRD consists of queries automatically generated from CNN/Daily Mail news articles; the answer to each query is a text span from a summarizing passage of the corresponding news. The goal of ReCoRD is to evaluate a machine's ability of commonsense reasoning in reading comprehension. + +## Few Shot evaluation +### Winogrande +#### Description +Winogrande is a benchmark for commonsense reasoning. The dataset contains pronoun resolution problems originally designed to be unsolvable for statistical models that rely on selectional preferences or word associations. + +### StoryCloze +#### Description +StoryCloze is a new commonsense reasoning framework for evaluating story understanding, story generation, and script learning. This test requires a system to choose the correct ending to a four-sentence story. + +### OpenBookQA +#### Description +OpenBookQA is a new kind of question-answering dataset modeled after open book exams for assessing human understanding of a subject. It consists of 5,957 multiple-choice elementary-level science questions (4,957 train, 500 dev, 500 test), which probe the understanding of a small “book” of 1,326 core science facts and the application of these facts to novel situations. + +## Fully supervised evaluation + +### BoolQ +#### Description +BoolQ is a question answering dataset for yes/no questions containing 15942 examples. These questions are naturally occurring – they are generated in unprompted and unconstrained settings. Each example is a triplet of (question, passage, answer), with the title of the page as optional additional context. + +### SST-2 +#### Description +SST-2 (or SST-binary) is a binary classification dataset where the goal is to differentiate between negative or somewhat negative vs somewhat positive or positive. + +### MNLI +#### Description +The Multi-Genre Natural Language Inference (MultiNLI) corpus is a crowd-sourced collection of 433k sentence pairs annotated with textual entailment information. The corpus is modeled on the SNLI corpus, but differs in that covers a range of genres of spoken and written text, and supports a distinctive cross-genre generalization evaluation. + +## Responsible AI (RAI) evaluation +### StereoSet +#### Description +A large-scale natural dataset in English to measure stereotypical biases in four domains: gender, profession, race, and religion + +#### Motivation for dataset use +The motivation for evaluating the 1.1T parameter model on this dataset is to evaluate the model's stereotype bias in gender, profession, race, and religion + +### CrowS +#### Description +Challenge Dataset for Measuring Social Biases in Masked Language Models + +#### Motivation for dataset use +The motivation for evaluating the 1.1T parameter model on this dataset is to evaluate the model’s bias in the domains of race, religion and age + +---- + +## Training data +### BookCorpus +#### Description +A dataset consisting of more than 10K unpublished books. 4GB in size. (Zhu et al., 2019) + +### English Wikipedia +#### Description +Data from English wikipedia, excluding lists, tables and headers. 12GB in size. + +### CC-News +#### Description +A dataset containing 63 millions English news articles crawled between September 2016 and February 2019. 76GB in size. (Nagel,2016) + +### OpenWebText +#### Description +An open source recreation of the WebText dataset used to train GPT-2. 38GB in size. (Gokaslan and Cohen, 2019) + +### CC-Stories +#### Description +A dataset containing a subset of CommonCrawl data filtered to match the story-like style of Winograd schemas. 31GB in size. (Trinh and Le, 2018) + +### English CC100 +#### Description +A dataset extracted from CommonCrawl snapshots between January 2018 and December 2018, filtered to match the style of Wikipedia following the methodology introduced in CCNet (https://arxiv.org/abs/1911.00359). 292GB in size. (Wenzek et al., 2020) + +## Responsible AI (RAI) Dimensions +### Fairness (Bias and inclusion) +The 1.1T parameter model was evaluated on the StereoSet and CrowS pairs dataset for inherent bias in the model, and bias as a result of the data. Similar to StereoSet, we observe that both the dense and MoE models get worse in terms of the Stereotype Score (SS) with scale. + +### Privacy and security +The 1.1T model did not have any special Privacy and Security considerations. The training data and evaluation data were both public and went through standard Meta privacy and licensing procedures. + +### Transparency and control +In the spirit of transparency and accountability we have created this model card for the 1.1T parameter model and a data card for the training data (referenced in Artetxe et al. (2021)). + +### Efficiency (Green AI) +The 1.1T parameter model is trained as a Mixture of Experts (MoE) model. Mixture of expert (MoE) models are efficient because they leverage sparse computation, i.e., only a small fraction of parameters are active for any given input. For instance, our 1.1T parameter MoE model requires only 30% more FLOPS compared to a 6.7B parameter dense model, i.e., a 160x increase in parameters with only a 30% increase in FLOPS. Notably, MoE models achieve much better validation perplexity for a given compute budget compared to dense models. + +## References +Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791– 4800, Florence, Italy. Association for Computational Linguistics. + +Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432–7439. + +Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint 1810.12885. + +Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8732–8740. + +Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839–849, San Diego, California. Association for Computational Linguistics. + +Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association for Computational Linguistics. + +Christopher Clark and Kenton Lee and Ming-Wei Chang and Tom Kwiatkowski and Michael Collins and Kristina Toutanova. 2019. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions + +Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Association for Computational Linguistics (ACL). + +Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. Association for Computational Linguistics. + +Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2019. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. arXiv:1506.06724. + +Sebastian Nagel. 2016. Cc-news. http: //web.archive.org/save/http: //commoncrawl.org/2016/10/news-dataset-available. + +Aaron Gokaslan and Vanya Cohen. 2019. Openwebtext corpus. http://web.archive.org/save/http://Skylion007.github.io/OpenWebTextCorpus + +Trieu H Trinh and Quoc V Le. 2018. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847. + +Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4003–4012, Marseille, France. European Language Resources Association. diff --git a/data/fairseq/examples/pay_less_attention_paper/README.md b/data/fairseq/examples/pay_less_attention_paper/README.md new file mode 100644 index 0000000000000000000000000000000000000000..5adab11f4dc3461f9e7126ac391b04e703616e6b --- /dev/null +++ b/data/fairseq/examples/pay_less_attention_paper/README.md @@ -0,0 +1,176 @@ +# Pay Less Attention with Lightweight and Dynamic Convolutions (Wu et al., 2019) + +This page contains pointers to pre-trained models as well as instructions on how to train new models for [our paper](https://arxiv.org/abs/1901.10430). + +## Citation: +```bibtex +@inproceedings{wu2018pay, + title = {Pay Less Attention with Lightweight and Dynamic Convolutions}, + author = {Felix Wu and Angela Fan and Alexei Baevski and Yann Dauphin and Michael Auli}, + booktitle = {International Conference on Learning Representations}, + year = {2019}, + url = {https://arxiv.org/abs/1901.10430}, +} +``` + +## Translation + +### Pre-trained models +For some datasets we release models without GLUs which are faster at inference. + +Model | Description | Dataset | Download +---|---|---|--- +`lightconv.no_glu.iwslt14.de-en` | LightConv (without GLUs) | [IWSLT14 German-English](https://wit3.fbk.eu/archive/2014-01/texts/de/en/de-en.tgz) | model:
[download (.tar.gz)](https://dl.fbaipublicfiles.com/fairseq/models/dynamicconv/iwslt14.de-en.lightconv.tar.gz)
IWSLT14 test:
[download (.tar.bz2)](https://dl.fbaipublicfiles.com/fairseq/data/iwslt14.de-en.test.tar.bz2) +`dynamicconv.no_glu.iwslt14.de-en` | DynamicConv (without GLUs) | [IWSLT14 German-English](https://wit3.fbk.eu/archive/2014-01/texts/de/en/de-en.tgz) | model:
[download (.tar.gz)](https://dl.fbaipublicfiles.com/fairseq/models/dynamicconv/iwslt14.de-en.dynamicconv.tar.gz)
IWSLT14 test:
[download (.tar.bz2)](https://dl.fbaipublicfiles.com/fairseq/data/iwslt14.de-en.test.tar.bz2) +`lightconv.no_glu.wmt16.en-de` | LightConv (without GLUs) | [WMT16 English-German](https://drive.google.com/uc?export=download&id=0B_bZck-ksdkpM25jRUN2X2UxMm8) | model:
[download (.tar.gz)](https://dl.fbaipublicfiles.com/fairseq/models/dynamicconv/wmt16.en-de.joined-dict.lightconv.tar.gz)
newstest2014 (shared vocab):
[download (.tar.bz2)](https://dl.fbaipublicfiles.com/fairseq/data/wmt16.en-de.joined-dict.newstest2014.tar.bz2) +`dynamicconv.no_glu.wmt16.en-de` | DynamicConv (without GLUs) | [WMT16 English-German](https://drive.google.com/uc?export=download&id=0B_bZck-ksdkpM25jRUN2X2UxMm8) | model:
[download (.tar.gz)](https://dl.fbaipublicfiles.com/fairseq/models/dynamicconv/wmt16.en-de.joined-dict.dynamicconv.tar.gz)
newstest2014 (shared vocab):
[download (.tar.bz2)](https://dl.fbaipublicfiles.com/fairseq/data/wmt16.en-de.joined-dict.newstest2014.tar.bz2) +`lightconv.glu.wmt16.en-de` | LightConv | [WMT16 English-German](https://drive.google.com/uc?export=download&id=0B_bZck-ksdkpM25jRUN2X2UxMm8) | model:
[download (.tar.gz)](https://dl.fbaipublicfiles.com/fairseq/models/dynamicconv/wmt16.en-de.joined-dict.lightconv-glu.tar.gz)
newstest2014 (shared vocab):
[download (.tar.bz2)](https://dl.fbaipublicfiles.com/fairseq/data/wmt16.en-de.joined-dict.newstest2014.tar.bz2) +`dynamicconv.glu.wmt16.en-de` | DynamicConv | [WMT16 English-German](https://drive.google.com/uc?export=download&id=0B_bZck-ksdkpM25jRUN2X2UxMm8) | model:
[download (.tar.gz)](https://dl.fbaipublicfiles.com/fairseq/models/dynamicconv/wmt16.en-de.joined-dict.dynamicconv-glu.tar.gz)
newstest2014 (shared vocab):
[download (.tar.bz2)](https://dl.fbaipublicfiles.com/fairseq/data/wmt16.en-de.joined-dict.newstest2014.tar.bz2) +`lightconv.glu.wmt14.en-fr` | LightConv | [WMT14 English-French](http://statmt.org/wmt14/translation-task.html#Download) | model:
[download (.tar.gz)](https://dl.fbaipublicfiles.com/fairseq/models/dynamicconv/wmt14.en-fr.joined-dict.lightconv-glu.tar.gz)
newstest2014:
[download (.tar.bz2)](https://dl.fbaipublicfiles.com/fairseq/data/wmt14.en-fr.joined-dict.newstest2014.tar.bz2) +`dynamicconv.glu.wmt14.en-fr` | DynamicConv | [WMT14 English-French](http://statmt.org/wmt14/translation-task.html#Download) | model:
[download (.tar.gz)](https://dl.fbaipublicfiles.com/fairseq/models/dynamicconv/wmt14.en-fr.joined-dict.dynamicconv-glu.tar.gz)
newstest2014:
[download (.tar.bz2)](https://dl.fbaipublicfiles.com/fairseq/data/wmt14.en-fr.joined-dict.newstest2014.tar.bz2) +`lightconv.glu.wmt17.zh-en` | LightConv | [WMT17 Chinese-English](http://statmt.org/wmt17/translation-task.html#Download) | model:
[download (.tar.gz)](https://dl.fbaipublicfiles.com/fairseq/models/dynamicconv/wmt17.zh-en.lightconv-glu.tar.gz)
newstest2017:
[download (.tar.bz2)](https://dl.fbaipublicfiles.com/fairseq/data/wmt17.zh-en.newstest2017.tar.bz2) +`dynamicconv.glu.wmt17.zh-en` | DynamicConv | [WMT17 Chinese-English](http://statmt.org/wmt17/translation-task.html#Download) | model:
[download (.tar.gz)](https://dl.fbaipublicfiles.com/fairseq/models/dynamicconv/wmt17.zh-en.dynamicconv-glu.tar.gz)
newstest2017:
[download (.tar.bz2)](https://dl.fbaipublicfiles.com/fairseq/data/wmt17.zh-en.newstest2017.tar.bz2) + +### Memory-Efficient CUDA Kernels + +Since the PyTorch implementations of Light/Dynamic conv are quite memory intensive, we have developed CUDA kernels that implement the light and dynamic convolution operator in a memory-efficient and performant manner. For large sequence lengths, these kernels save about 50% memory compared to the PyTorch equivalent. + +To install the kernels, use the commands below. Once installed, they will automatically be used in place of the PyTorch implementations whenever a light or dynamic convolution is used. + +```sh +# to install lightconv +cd fairseq/modules/lightconv_layer +python cuda_function_gen.py +python setup.py install + +# to install dynamicconv +cd fairseq/modules/dynamicconv_layer +python cuda_function_gen.py +python setup.py install +``` + +### Example usage (torch.hub) + +We require a few additional Python dependencies for preprocessing: +```bash +pip install sacremoses subword_nmt +``` + +Interactive translation via PyTorch Hub: +```python +import torch + +# List available models +torch.hub.list('pytorch/fairseq') # [..., 'lightconv.glu.wmt17.zh-en', ... ] + +# Load a transformer trained on WMT'16 En-De +zh2en = torch.hub.load('pytorch/fairseq', 'lightconv.glu.wmt17.zh-en', tokenizer='moses', bpe='subword_nmt') + +# The underlying model is available under the *models* attribute +assert isinstance(zh2en.models[0], fairseq.models.lightconv.LightConvModel) + +# Translate a sentence +zh2en.translate('你好 世界') +# 'Hello World' +``` + +Loading custom models: +```python +from fairseq.models.lightconv import LightConvModel +en2fr = LightConvModel.from_pretrained( + '/path/to/checkpoints', + checkpoint_file='checkpoint_best.pt', + data_name_or_path='data-bin/wmt14_en_fr', + bpe='subword_nmt', + bpe_codes='data-bin/wmt14_en_fr/en.code' +) +en2fr.translate('Hello world!') +# 'Bonjour le monde' +``` + +### Preprocessing the training datasets + +Please follow the instructions in [`examples/translation/README.md`](../translation/README.md) to preprocess the data. + +### Training and evaluation options: +To use the model without GLU, please set `--encoder-glu 0 --decoder-glu 0`. +For LightConv, please use `--encoder-conv-type lightweight --decoder-conv-type lightweight`, otherwise the default is DynamicConv. +For best BLEU results, lenpen may need to be manually tuned. + +To use the CUDA kernels, first install the PyTorch modules using the commands +above. Once the CUDA modules are installed, they will automatically be used +instead of the PyTorch modules. + +### IWSLT14 De-En +Training and evaluating DynamicConv (without GLU) on a GPU: +```sh +# Training +SAVE="save/dynamic_conv_iwslt" +mkdir -p $SAVE +CUDA_VISIBLE_DEVICES=0 $(which fairseq-train) data-bin/iwslt14.tokenized.de-en \ + --clip-norm 0 --optimizer adam --lr 0.0005 \ + --source-lang de --target-lang en --max-tokens 4000 --no-progress-bar \ + --log-interval 100 --stop-min-lr '1e-09' --weight-decay 0.0001 \ + --criterion label_smoothed_cross_entropy --label-smoothing 0.1 \ + --lr-scheduler inverse_sqrt \ + --ddp-backend=legacy_ddp \ + --max-update 50000 --warmup-updates 4000 --warmup-init-lr '1e-07' \ + --adam-betas '(0.9, 0.98)' --keep-last-epochs 10 \ + -a lightconv_iwslt_de_en --save-dir $SAVE \ + --dropout 0.3 --attention-dropout 0.1 --weight-dropout 0.1 \ + --encoder-glu 0 --decoder-glu 0 +python scripts/average_checkpoints.py --inputs $SAVE \ + --num-epoch-checkpoints 10 --output "${SAVE}/checkpoint_last10_avg.pt" + +# Evaluation +CUDA_VISIBLE_DEVICES=0 fairseq-generate data-bin/iwslt14.tokenized.de-en --path "${SAVE}/checkpoint_last10_avg.pt" --batch-size 128 --beam 4 --remove-bpe --lenpen 1 --gen-subset test --quiet +``` + +### WMT16 En-De +Training and evaluating DynamicConv (with GLU) on WMT16 En-De using cosine scheduler on one machine with 8 V100 GPUs: +```sh +# Training +SAVE="save/dynamic_conv_wmt16en2de" +mkdir -p $SAVE +python -m torch.distributed.launch --nproc_per_node 8 $(which fairseq-train) \ + data-bin/wmt16_en_de_bpe32k --fp16 --log-interval 100 --no-progress-bar \ + --max-update 30000 --share-all-embeddings --optimizer adam \ + --adam-betas '(0.9, 0.98)' --clip-norm 0.0 --weight-decay 0.0 \ + --criterion label_smoothed_cross_entropy --label-smoothing 0.1 \ + --stop-min-lr 1e-09 --update-freq 16 --attention-dropout 0.1 --keep-last-epochs 10 \ + --ddp-backend=legacy_ddp --max-tokens 3584 \ + --lr-scheduler cosine --warmup-init-lr 1e-7 --warmup-updates 10000 \ + --lr-shrink 1 --lr 0.001 --min-lr 1e-7 --warmup-init-lr 1e-07 \ + --t-mult 1 --lr-period-updates 20000 \ + --arch lightconv_wmt_en_de_big --save-dir $SAVE \ + --dropout 0.3 --attention-dropout 0.1 --weight-dropout 0.1 \ + --encoder-glu 1 --decoder-glu 1 + +# Evaluation +CUDA_VISIBLE_DEVICES=0 fairseq-generate data-bin/wmt16.en-de.joined-dict.newstest2014 --path "${SAVE}/checkpoint_best.pt" --batch-size 128 --beam 5 --remove-bpe --lenpen 0.5 --gen-subset test > wmt16_gen.txt +bash scripts/compound_split_bleu.sh wmt16_gen.txt +``` + +### WMT14 En-Fr +Training DynamicConv (with GLU) on WMT14 En-Fr using cosine scheduler on one machine with 8 V100 GPUs: +```sh +# Training +SAVE="save/dynamic_conv_wmt14en2fr" +mkdir -p $SAVE +python -m torch.distributed.launch --nproc_per_node 8 $(which fairseq-train) \ + data-bin/wmt14_en_fr --fp16 --log-interval 100 --no-progress-bar \ + --max-update 30000 --share-all-embeddings --optimizer adam \ + --adam-betas '(0.9, 0.98)' --clip-norm 0.0 --weight-decay 0.0 \ + --criterion label_smoothed_cross_entropy --label-smoothing 0.1 \ + --stop-min-lr 1e-09 --update-freq 16 --attention-dropout 0.1 --keep-last-epochs 10 \ + --ddp-backend=legacy_ddp --max-tokens 3584 \ + --lr-scheduler cosine --warmup-init-lr 1e-7 --warmup-updates 10000 \ + --lr-shrink 1 --lr 0.001 --min-lr 1e-7 --warmup-init-lr 1e-07 \ + --t-mult 1 --lr-period-updates 70000 \ + --arch lightconv_wmt_en_fr_big --save-dir $SAVE \ + --dropout 0.1 --attention-dropout 0.1 --weight-dropout 0.1 \ + --encoder-glu 1 --decoder-glu 1 + +# Evaluation +CUDA_VISIBLE_DEVICES=0 fairseq-generate data-bin/wmt14.en-fr.joined-dict.newstest2014 --path "${SAVE}/checkpoint_best.pt" --batch-size 128 --beam 5 --remove-bpe --lenpen 0.9 --gen-subset test +``` diff --git a/data/fairseq/examples/quant_noise/README.md b/data/fairseq/examples/quant_noise/README.md new file mode 100644 index 0000000000000000000000000000000000000000..a04d7e4e8a077f11c9f63cfa3d1f20e2b899be8c --- /dev/null +++ b/data/fairseq/examples/quant_noise/README.md @@ -0,0 +1,298 @@ +# Training with Quantization Noise for Extreme Model Compression ({Fan\*, Stock\*} *et al.*, 2020) +This page contains information for how to train and quantize models with Quantization Noise, for both scalar quantization like `int8` and Iterative Product Quantization. +Check out our paper [here](https://arxiv.org/abs/2004.07320). + +Looking for pretrained models? They will be added shortly. +Looking for code to train vision models? We are working on open sourcing our code as part of ClassyVision. Please check back, but note that both the Scalar and Iterative Product Quantization counterparts of the `nn.Conv2d` module are already included in this release. + +**Contents**: +- [Walk through of code](#walk-through-the-code) +- [Reproduce NLP Results](#looking-to-reproduce-the-nlp-results-in-the-paper) +- [Reproduce Vision Results](#looking-to-reproduce-the-vision-results-in-the-paper) + + +## Citation +```bibtex +@article{fan2020training, + title={Training with Quantization Noise for Extreme Model Compression}, + author={Angela Fan* and Pierre Stock* and and Benjamin Graham and Edouard Grave and Remi Gribonval and Herve Jegou and Armand Joulin}, + year={2020}, + eprint={2004.07320}, + archivePrefix={arXiv}, + primaryClass={cs.ML} +} +``` + +## Walk through the code + +Training a model with Quant-Noise improves the performance in subsequent inference-time quantization by training models to be robust to quantization. This technique is useful for both scalar and product quantization methods, as well as multiple domains. We detail below our approach to train, quantize models and integrate our code to quantize your favorite models. + +### Scalar Quantization + +Unlike the section [Iterative Product Quantization](#iterative-product-quantization) which gives state-of-the-art compression, this section showcases the usefulness of our approach for simple scalar quantization baselines such as int8 using on-GPU Fake Quantization. + +#### Training + +Scalar quantization with Quant-Noise consists in randomly quantizing a proportion `p` of the weights during training. Scalar quantization is implemented [here](https://github.com/pytorch/fairseq/tree/main/fairseq/modules/quantization/scalar) under the form of Fake Quantization, meaning that we emulate int8 on GPU by quantizing and de-quantizing both the weights and the activations. We rely on PyTorch's [quantization primitives](https://github.com/pytorch/pytorch/tree/master/torch/quantization). + +To train a model with Quant-Noise, add the following flag: +``` +--quant-noise-scalar 0.5 +``` +Large values of noise make the network easier to quantize but may result in higher non-quantized test and validation perplexities. + +#### Quantization + +When evaluating a network, all quantized modules and activation hooks automatically switch to `p=1` so the validation accuracy reported by Fairseq is actually the quantized one, nothing more to do. + + +#### Integration with your own code + +Looking to quantize your own models with Quant-Noise + Scalar Quantization? +- Use the function `quantize_model_` implemented [here](https://github.com/pytorch/fairseq/tree/main/fairseq/modules/quantization/scalar/utils.py) to (1) replace all your modules by their quantized counterparts and (2) add hooks to those modules to quantize the activations. +- Then, perform your training as usual. Note that in `eval()` mode, the network is always fully quantized (weights and activations) by default (`p=1`). + + + +### Iterative Product Quantization + + +Iterative Product Quantization with Quant-Noise proceeds in two steps. First, a model must be trained uncompressed with Quant-Noise. Second, the model must be quantized with iPQ. Note that we implement here the simplest form of noise, which consists in randomly dropping a proportion `p` of blocks, and that worked as well as assigning those blocks to their current centroid. + +#### Training + +To train a model with Quant-Noise, add the following flags: +``` +--quant-noise-pq 0.1 --quant-noise-pq-block-size 8 +``` +`quant-noise-pq` controls how much dropout is applied to the blocks of the weight matrix. `quant-noise-pq-block-size` controls the size of the weight matrix blocks. +We recommend training with 0.05 to 0.2 Quant-Noise, a value that worked well in our experiments. For the block-size, we recommend training with block-size of 8. Note that the block size must be a multiple of `input_features`, see the size checks [here](https://github.com/pytorch/fairseq/tree/main/fairseq/modules/quant_noise.py). Large block sizes result in higher compression ratio but may induce a loss in accuracy. + +We currently support training Transformer based models, such as sequence-to-sequence, language models, and BERT architectures. The `quant_noise` function [here](https://github.com/pytorch/fairseq/tree/main/fairseq/modules/quant_noise.py) wraps a module. It splits a weight matrix into blocks and applies random dropout to these blocks. +In the Transformer architectures, quant-noise is applied to the input and output embeddings, the attention, and the FFN. + +Quant-Noise can also be combined with **LayerDrop** (see [here](https://github.com/pytorch/fairseq/tree/main/examples/layerdrop)) to add its pruning effect to the quantized model and make the model even smaller. We recommend training with LayerDrop 0.1 or 0.2. + +#### Quantization + +We implement an improved version of product quantization from Stock et al, **iPQ**, described [here](https://arxiv.org/abs/1907.05686), see code with old API [here](https://github.com/facebookresearch/kill-the-bits). Note that we improved the iPQ API in terms of both compute speed and usability as described below. + +For the particular case of PQ, quantization is made sequentially. We recommend first quantizing the FFNs, then the EMBs, and finally the ATTNs. Quantization is done in two sub-steps: +- First, perform `n` steps of Product Quantization (generally `n=20` is enough). +- Then, finetune the obtained centroids. + +#### Integration with your own code + +Looking to quantize your own models with Quant-Noise + iPQ? +- First wrap your modules with the `quant_noise` function [here](https://github.com/pytorch/fairseq/tree/main/fairseq/modules/quant_noise.py), which is module-agnostic and train your favorite model. +- Then, quantize your trained model using the code [here](https://github.com/pytorch/fairseq/tree/main/fairseq/modules/quantization/pq). This can be done *without any changes to your training loop*. Below is an example code for integration. +Note that we tried our approach only on Transformers and various Convolutional Models such as EfficientNets. + +```python +from fairseq.modules.quantization.pq import quantize_model_, SizeTracker + +# get configuration parameters +n_centroids_config = config["n_centroids"] +block_sizes_config = config["block_sizes"] +layers_to_quantize = config["layers_to_quantize"] + +# size tracker for keeping track of assignments, centroids and non-compressed sizes +size_tracker = SizeTracker(model) + +# Quantize model by stages +for step in range(len(layers_to_quantize)): + + # quantize model in-place + quantized_layers = quantize_model_( + model, + size_tracker, + layers_to_quantize, + block_sizes_config, + n_centroids_config, + step=step, + ) + logger.info(f"Finetuning stage {step}, quantized layers: {quantized_layers}") + logger.info(f"{size_tracker}") + + # Don't forget to re-create/update trainer/optimizer since model parameters have changed + optimizer = ... + + # Finetune the centroids with your usual training loop for a few epochs + trainer.train_epoch() +``` + + +## Looking to reproduce the NLP results in the paper? + +We detail below how to reproduce the state-of-the-art results in reported in the paper for Quant-Noise + Iterative Product Quantization. + +### Training with Quant-Noise + +To **train** RoBERTa + QuantNoise, we followed this setting [here](https://github.com/pytorch/fairseq/tree/main/examples/roberta). +The following command can be used to train a RoBERTa Base + QuantNoise model: + +```bash +TOTAL_UPDATES=125000 +WARMUP_UPDATES=10000 +PEAK_LR=0.0005 +TOKENS_PER_SAMPLE=512 +MAX_POSITIONS=512 +MAX_SENTENCES=16 +UPDATE_FREQ=2 +DATA_DIR=/path/to/data/here + +fairseq-train $DATA_DIR \ + --task masked_lm --criterion masked_lm --arch roberta_base \ + --sample-break-mode complete \ + --tokens-per-sample $TOKENS_PER_SAMPLE --max-positions $MAX_POSITIONS \ + --optimizer adam --adam-betas '(0.9, 0.98)' --adam-eps 1e-6 \ + --clip-norm 0.0 \ + --lr-scheduler polynomial_decay --lr $PEAK_LR \ + --warmup-updates $WARMUP_UPDATES --total-num-update $TOTAL_UPDATES \ + --dropout 0.1 --attention-dropout 0.1 \ + --weight-decay 0.01 \ + --batch-size $MAX_SENTENCES \ + --update-freq $UPDATE_FREQ --max-update $TOTAL_UPDATES \ + --save-dir checkpoint/roberta \ + --ddp-backend legacy_ddp --encoder-layerdrop 0.2 \ + --quant-noise-pq 0.2 --quant-noise-pq-block-size 8 --untie-weights-roberta +``` + +To **finetune** RoBERTa + QuantNoise, we followed this setting [here](https://github.com/pytorch/fairseq/blob/main/examples/roberta/README.glue.md). +The following command can be used to finetune a RoBERTa Base + QuantNoise model on the RTE dataset: + +```bash +TOTAL_NUM_UPDATES=2036 +WARMUP_UPDATES=122 +LR=2e-05 +NUM_CLASSES=2 +MAX_SENTENCES=16 +ROBERTA_PATH=/path/to/roberta_quantnoise/model.pt + +fairseq-train /path/to/rte/data/ \ + --restore-file $ROBERTA_PATH \ + --max-positions 512 \ + --batch-size $MAX_SENTENCES \ + --max-tokens 4400 \ + --task sentence_prediction \ + --reset-optimizer --reset-dataloader --reset-meters \ + --required-batch-size-multiple 1 \ + --init-token 0 --separator-token 2 \ + --arch roberta_large \ + --criterion sentence_prediction \ + --num-classes $NUM_CLASSES \ + --dropout 0.1 --attention-dropout 0.1 \ + --weight-decay 0.1 --optimizer adam --adam-betas "(0.9, 0.98)" --adam-eps 1e-06 \ + --clip-norm 0.0 \ + --lr-scheduler polynomial_decay --lr $LR --total-num-update $TOTAL_NUM_UPDATES --warmup-updates $WARMUP_UPDATES \ + --fp16 --fp16-init-scale 4 --threshold-loss-scale 1 --fp16-scale-window 128 \ + --max-epoch 10 \ + --find-unused-parameters \ + --best-checkpoint-metric accuracy --maximize-best-checkpoint-metric \ + --ddp-backend legacy_ddp \ + --quant-noise-pq 0.2 --quant-noise-pq-block-size 8 +``` + +To **train** Language Models on Wikitext-103, we followed this setting [here](https://github.com/pytorch/fairseq/tree/main/examples/language_model). +The following command can be used to train a Transformer + QuantNoise model on Wikitext-103: + +```bash +fairseq-train --task language_modeling /path/to/wikitext-103/data \ + --save-dir checkpoints/transformer_wikitext-103 \ + --adaptive-input --adaptive-input-cutoff 20000,60000 --adaptive-input-factor 4 \ + --adaptive-softmax-cutoff 20000,60000 --adaptive-softmax-dropout 0.2 --adaptive-softmax-factor 4.0 \ + --tie-adaptive-proj --tie-adaptive-weights \ + --arch transformer_lm_gbw \ + --attention-dropout 0.1 --dropout 0.2 --relu-dropout 0.1 \ + --clip-norm 0.1 --criterion adaptive_loss \ + --ddp-backend legacy_ddp \ + --decoder-attention-heads 8 --decoder-embed-dim 1024 --decoder-ffn-embed-dim 4096 --decoder-input-dim 1024 \ + --decoder-layers 16 --decoder-normalize-before --decoder-output-dim 1024 \ + --min-lr 0.0001 --lr-period-updates 270000 --lr-scheduler cosine --lr-shrink 0.75 --lr 1.0 --t-mult 2.0 \ + --max-tokens 3072 --tokens-per-sample 3072 --momentum 0.99 --optimizer nag \ + --sample-break-mode none --update-freq 3 \ + --warmup-init-lr 1e-07 --warmup-updates 16000 \ + --weight-decay 0 --seed 1 --stop-min-lr 1e-09 \ + --quant-noise-pq 0.05 --quant-noise-pq-block-size 8 +``` + +To **evaluate** this model, note you need to use the `eval.py` script. The following command can be used to evaluate: + +```bash +fairseq-eval-lm /path/to/wikitext-103/data --path /path/to/model/checkpoint \ + --sample-break-mode complete \ + --max-tokens 3072 \ + --context-window 2560 \ + --softmax-batch 1024 \ + --gen-subset valid +``` +and change the `--gen-subset` to `test` if you would like to evaluate on the test set instead. + + +### Iterative Product Quantization + +To quantize the finetuned RoBERTa model, we use this command on 1 GPU. This should run in a day. +```bash +TOTAL_NUM_UPDATES=6108 # 2036 updates for each iteration +WARMUP_UPDATES=122 +LR=2e-05 +NUM_CLASSES=2 +MAX_SENTENCES=16 +fairseq-train --task sentence_prediction /path/to/data/ \ + --restore-file $ROBERTA_PATH \ + --save-dir checkpoints/roberta_finetuned \ + --max-positions 512 \ + --batch-size $MAX_SENTENCES \ + --max-tokens 4400 \ + --init-token 0 --separator-token 2 \ + --arch roberta_large \ + --criterion sentence_prediction \ + --num-classes $NUM_CLASSES \ + --dropout 0.1 --attention-dropout 0.1 \ + --weight-decay 0.1 --optimizer adam --adam-betas "(0.9, 0.98)" --adam-eps 1e-06 \ + --clip-norm 0.0 --lr-scheduler polynomial_decay \ + --fp16 --fp16-init-scale 4 --threshold-loss-scale 1 --fp16-scale-window 128 \ + --no-progress-bar --skip-invalid-size-inputs-valid-test --ddp-backend legacy_ddp \ + --quantization-config-path /path/to/config/yaml +``` + +To quantize the trained Language Model, we use this command on 8 V100 23GB GPUs. This should run in a couple of hours. +```bash +fairseq-train --task language_modeling /path/to/wikitext-103/data \ + --save-dir checkpoints/transformer_wikitext-103 \ + --adaptive-input --adaptive-input-cutoff 20000,60000 --adaptive-input-factor 4 \ + --adaptive-softmax-cutoff 20000,60000 --adaptive-softmax-dropout 0.2 --adaptive-softmax-factor 4.0 \ + --arch transformer_lm_gbw \ + --attention-dropout 0.1 --dropout 0.2 --relu-dropout 0.1 \ + --bucket-cap-mb 25 --char-embedder-highway-layers 2 --character-embedding-dim 4 \ + --clip-norm 0.1 --criterion adaptive_loss \ + --ddp-backend legacy_ddp \ + --decoder-attention-heads 8 --decoder-embed-dim 1024 --decoder-ffn-embed-dim 4096 --decoder-input-dim 1024 --decoder-layers 16 --decoder-normalize-before --decoder-output-dim 1024 \ + --fp16 --keep-last-epochs -1 \ + --min-lr 0.0001 --lr-period-updates 270000 --lr-scheduler cosine --lr-shrink 0.75 --lr 0.05 --stop-min-lr 1e-09 \ + --max-tokens 2944 --tokens-per-sample 2944\ + --momentum 0.99 --no-epoch-checkpoints --no-progress-bar --optimizer nag --required-batch-size-multiple 8 \ + --sample-break-mode none --t-mult 2.0 --skip-invalid-size-inputs-valid-test \ + --tie-adaptive-proj --tie-adaptive-weights --update-freq 3 --weight-decay 0 --seed 1 \ + --log-interval 100 --no-progress-bar --skip-invalid-size-inputs-valid-test \ + --restore-file path/to/trained/lm/with/quant/noise \ + --max-update 13500 --quantization-config-path /path/to/config/yaml +``` +If you have less capacity or if your distributed training freezes, try reducing `--max-tokens` and `--tokens-per-sample` (this may reduce the quantized accuracy a bit). + +### Remarks + +We try to keep the open-sourced code as readable and as easy-to-plug as possible. Therefore, we did not test it for the following cases: +- Scalar quantization with RoBERTa. +- Quantization with iPQ and `int8` combined. + +If you have trouble adapting it, we will be more than happy to help! + +## Looking to reproduce the Vision results in the paper? + +We are working on open sourcing our code as part of ClassyVision. Please check back. + + +## Having an issue or have a question? + +Please open an issue in this repository with the details of your question. Thanks! diff --git a/data/fairseq/examples/quant_noise/transformer_quantization_config.yaml b/data/fairseq/examples/quant_noise/transformer_quantization_config.yaml new file mode 100644 index 0000000000000000000000000000000000000000..d4be14a93a3593f8e6dc66c3b05061bfdde3e0e0 --- /dev/null +++ b/data/fairseq/examples/quant_noise/transformer_quantization_config.yaml @@ -0,0 +1,33 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +# This file defines example configuration arguments for quantizing +# a transformer model with product quantization + +# Number of Centroids for Product Quantization, by default 256 (byte-aligned) +n_centroids: + Linear: + key: in_features + value: {"*": 256} + Embedding: + key: embedding_dim + value: {"*": 256} + +# Block Sizes for Product Quantization +# We suggest: 8 for FFN, 4 for ATTN, 4 for embedding projections, 8 for embeddings +block_sizes: + Linear: + key: fuzzy_name + value: {fc: 8, attn: 4, emb: 4} + Embedding: + key: fuzzy_name + value: {emb: 8} + +# Layers to Quantize Sequentially +# We suggest: first FFN, then EMB, then ATTN +layers_to_quantize: + - decoder\\.layers\\.\d+\\.fc[12] + - decoder\\.embed_tokens\\.embeddings\\.[012]\\.[01] + - decoder\\.layers\\.\d+\\.self_attn\\.(k_proj|v_proj|q_proj|out_proj) diff --git a/data/fairseq/examples/speech_recognition/README.md b/data/fairseq/examples/speech_recognition/README.md new file mode 100644 index 0000000000000000000000000000000000000000..5f9b27880e439dc1803db7f6f0765edaefeb7097 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/README.md @@ -0,0 +1,87 @@ +### 2021 Update: We are merging this example into the [S2T framework](../speech_to_text), which supports more generic speech-to-text tasks (e.g. speech translation) and more flexible data processing pipelines. Please stay tuned. + +# Speech Recognition +`examples/speech_recognition` is implementing ASR task in Fairseq, along with needed features, datasets, models and loss functions to train and infer model described in [Transformers with convolutional context for ASR (Abdelrahman Mohamed et al., 2019)](https://arxiv.org/abs/1904.11660). + + +## Additional dependencies +On top of main fairseq dependencies there are couple more additional requirements. + +1) Please follow the instructions to install [torchaudio](https://github.com/pytorch/audio). This is required to compute audio fbank features. +2) [Sclite](http://www1.icsi.berkeley.edu/Speech/docs/sctk-1.2/sclite.htm#sclite_name_0) is used to measure WER. Sclite can be downloaded and installed from source from sctk package [here](http://www.openslr.org/4/). Training and inference doesn't require Sclite dependency. +3) [sentencepiece](https://github.com/google/sentencepiece) is required in order to create dataset with word-piece targets. + +## Preparing librispeech data +``` +./examples/speech_recognition/datasets/prepare-librispeech.sh $DIR_TO_SAVE_RAW_DATA $DIR_FOR_PREPROCESSED_DATA +``` + +## Training librispeech data +``` +python train.py $DIR_FOR_PREPROCESSED_DATA --save-dir $MODEL_PATH --max-epoch 80 --task speech_recognition --arch vggtransformer_2 --optimizer adadelta --lr 1.0 --adadelta-eps 1e-8 --adadelta-rho 0.95 --clip-norm 10.0 --max-tokens 5000 --log-format json --log-interval 1 --criterion cross_entropy_acc --user-dir examples/speech_recognition/ +``` + +## Inference for librispeech +`$SET` can be `test_clean` or `test_other` +Any checkpoint in `$MODEL_PATH` can be selected. In this example we are working with `checkpoint_last.pt` +``` +python examples/speech_recognition/infer.py $DIR_FOR_PREPROCESSED_DATA --task speech_recognition --max-tokens 25000 --nbest 1 --path $MODEL_PATH/checkpoint_last.pt --beam 20 --results-path $RES_DIR --batch-size 40 --gen-subset $SET --user-dir examples/speech_recognition/ +``` + +## Inference for librispeech +``` +sclite -r ${RES_DIR}/ref.word-checkpoint_last.pt-${SET}.txt -h ${RES_DIR}/hypo.word-checkpoint_last.pt-${SET}.txt -i rm -o all stdout > $RES_REPORT +``` +`Sum/Avg` row from first table of the report has WER + +## Using flashlight (previously called [wav2letter](https://github.com/facebookresearch/wav2letter)) components +[flashlight](https://github.com/facebookresearch/flashlight) now has integration with fairseq. Currently this includes: + +* AutoSegmentationCriterion (ASG) +* flashlight-style Conv/GLU model +* flashlight's beam search decoder + +To use these, follow the instructions on [this page](https://github.com/flashlight/flashlight/tree/e16682fa32df30cbf675c8fe010f929c61e3b833/bindings/python) to install python bindings. **Flashlight v0.3.2** must be used to install the bindings. Running: +``` +git clone --branch v0.3.2 https://github.com/flashlight/flashlight +``` +will properly clone and check out this version. + +## Training librispeech data (flashlight style, Conv/GLU + ASG loss) +Training command: +``` +python train.py $DIR_FOR_PREPROCESSED_DATA --save-dir $MODEL_PATH --max-epoch 100 --task speech_recognition --arch w2l_conv_glu_enc --batch-size 4 --optimizer sgd --lr 0.3,0.8 --momentum 0.8 --clip-norm 0.2 --max-tokens 50000 --log-format json --log-interval 100 --num-workers 0 --sentence-avg --criterion asg_loss --asg-transitions-init 5 --max-replabel 2 --linseg-updates 8789 --user-dir examples/speech_recognition +``` + +Note that ASG loss currently doesn't do well with word-pieces. You should prepare a dataset with character targets by setting `nbpe=31` in `prepare-librispeech.sh`. + +## Inference for librispeech (flashlight decoder, n-gram LM) +Inference command: +``` +python examples/speech_recognition/infer.py $DIR_FOR_PREPROCESSED_DATA --task speech_recognition --seed 1 --nbest 1 --path $MODEL_PATH/checkpoint_last.pt --gen-subset $SET --results-path $RES_DIR --w2l-decoder kenlm --kenlm-model $KENLM_MODEL_PATH --lexicon $LEXICON_PATH --beam 200 --beam-threshold 15 --lm-weight 1.5 --word-score 1.5 --sil-weight -0.3 --criterion asg_loss --max-replabel 2 --user-dir examples/speech_recognition +``` + +`$KENLM_MODEL_PATH` should be a standard n-gram language model file. `$LEXICON_PATH` should be a flashlight-style lexicon (list of known words and their spellings). For ASG inference, a lexicon line should look like this (note the repetition labels): +``` +doorbell D O 1 R B E L 1 ▁ +``` +For CTC inference with word-pieces, repetition labels are not used and the lexicon should have most common spellings for each word (one can use sentencepiece's `NBestEncodeAsPieces` for this): +``` +doorbell ▁DOOR BE LL +doorbell ▁DOOR B E LL +doorbell ▁DO OR BE LL +doorbell ▁DOOR B EL L +doorbell ▁DOOR BE L L +doorbell ▁DO OR B E LL +doorbell ▁DOOR B E L L +doorbell ▁DO OR B EL L +doorbell ▁DO O R BE LL +doorbell ▁DO OR BE L L +``` +Lowercase vs. uppercase matters: the *word* should match the case of the n-gram language model (i.e. `$KENLM_MODEL_PATH`), while the *spelling* should match the case of the token dictionary (i.e. `$DIR_FOR_PREPROCESSED_DATA/dict.txt`). + +## Inference for librispeech (flashlight decoder, viterbi only) +Inference command: +``` +python examples/speech_recognition/infer.py $DIR_FOR_PREPROCESSED_DATA --task speech_recognition --seed 1 --nbest 1 --path $MODEL_PATH/checkpoint_last.pt --gen-subset $SET --results-path $RES_DIR --w2l-decoder viterbi --criterion asg_loss --max-replabel 2 --user-dir examples/speech_recognition +``` diff --git a/data/fairseq/examples/speech_recognition/__init__.py b/data/fairseq/examples/speech_recognition/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..0278f6a27340c7ff7e207d09348483d1b0d3a100 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/__init__.py @@ -0,0 +1 @@ +from . import criterions, models, tasks # noqa diff --git a/data/fairseq/examples/speech_recognition/criterions/ASG_loss.py b/data/fairseq/examples/speech_recognition/criterions/ASG_loss.py new file mode 100644 index 0000000000000000000000000000000000000000..41f50bbd70388ce723f2d316d4e9776bcd6be3c9 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/criterions/ASG_loss.py @@ -0,0 +1,170 @@ +#!/usr/bin/env python3 + +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import torch +from examples.speech_recognition.data.replabels import pack_replabels +from fairseq import utils +from fairseq.criterions import FairseqCriterion, register_criterion + + +@register_criterion("asg_loss") +class ASGCriterion(FairseqCriterion): + @staticmethod + def add_args(parser): + group = parser.add_argument_group("ASG Loss") + group.add_argument( + "--asg-transitions-init", + help="initial diagonal value of transition matrix", + type=float, + default=0.0, + ) + group.add_argument( + "--max-replabel", help="maximum # of replabels", type=int, default=2 + ) + group.add_argument( + "--linseg-updates", + help="# of training updates to use LinSeg initialization", + type=int, + default=0, + ) + group.add_argument( + "--hide-linseg-messages", + help="hide messages about LinSeg initialization", + action="store_true", + ) + + def __init__( + self, + task, + silence_token, + asg_transitions_init, + max_replabel, + linseg_updates, + hide_linseg_messages, + ): + from flashlight.lib.sequence.criterion import ASGLoss, CriterionScaleMode + + super().__init__(task) + self.tgt_dict = task.target_dictionary + self.eos = self.tgt_dict.eos() + self.silence = ( + self.tgt_dict.index(silence_token) + if silence_token in self.tgt_dict + else None + ) + self.max_replabel = max_replabel + + num_labels = len(self.tgt_dict) + self.asg = ASGLoss(num_labels, scale_mode=CriterionScaleMode.TARGET_SZ_SQRT) + self.asg.trans = torch.nn.Parameter( + asg_transitions_init * torch.eye(num_labels), requires_grad=True + ) + + self.linseg_progress = torch.nn.Parameter( + torch.tensor([0], dtype=torch.int), requires_grad=False + ) + self.linseg_maximum = linseg_updates + self.linseg_message_state = "none" if hide_linseg_messages else "start" + + @classmethod + def build_criterion(cls, args, task): + return cls( + task, + args.silence_token, + args.asg_transitions_init, + args.max_replabel, + args.linseg_updates, + args.hide_linseg_messages, + ) + + def linseg_step(self): + if not self.training: + return False + if self.linseg_progress.item() < self.linseg_maximum: + if self.linseg_message_state == "start": + print("| using LinSeg to initialize ASG") + self.linseg_message_state = "finish" + self.linseg_progress.add_(1) + return True + elif self.linseg_message_state == "finish": + print("| finished LinSeg initialization") + self.linseg_message_state = "none" + return False + + def replace_eos_with_silence(self, tgt): + if tgt[-1] != self.eos: + return tgt + elif self.silence is None or (len(tgt) > 1 and tgt[-2] == self.silence): + return tgt[:-1] + else: + return tgt[:-1] + [self.silence] + + def forward(self, model, sample, reduce=True): + """Compute the loss for the given sample. + + Returns a tuple with three elements: + 1) the loss + 2) the sample size, which is used as the denominator for the gradient + 3) logging outputs to display while training + """ + + net_output = model(**sample["net_input"]) + emissions = net_output["encoder_out"].transpose(0, 1).contiguous() + B = emissions.size(0) + T = emissions.size(1) + device = emissions.device + + target = torch.IntTensor(B, T) + target_size = torch.IntTensor(B) + using_linseg = self.linseg_step() + + for b in range(B): + initial_target_size = sample["target_lengths"][b].item() + if initial_target_size == 0: + raise ValueError("target size cannot be zero") + + tgt = sample["target"][b, :initial_target_size].tolist() + tgt = self.replace_eos_with_silence(tgt) + tgt = pack_replabels(tgt, self.tgt_dict, self.max_replabel) + tgt = tgt[:T] + + if using_linseg: + tgt = [tgt[t * len(tgt) // T] for t in range(T)] + + target[b][: len(tgt)] = torch.IntTensor(tgt) + target_size[b] = len(tgt) + + loss = self.asg.forward(emissions, target.to(device), target_size.to(device)) + + if reduce: + loss = torch.sum(loss) + + sample_size = ( + sample["target"].size(0) if self.args.sentence_avg else sample["ntokens"] + ) + logging_output = { + "loss": utils.item(loss.data) if reduce else loss.data, + "ntokens": sample["ntokens"], + "nsentences": sample["target"].size(0), + "sample_size": sample_size, + } + return loss, sample_size, logging_output + + @staticmethod + def aggregate_logging_outputs(logging_outputs): + """Aggregate logging outputs from data parallel training.""" + loss_sum = sum(log.get("loss", 0) for log in logging_outputs) + ntokens = sum(log.get("ntokens", 0) for log in logging_outputs) + nsentences = sum(log.get("nsentences", 0) for log in logging_outputs) + sample_size = sum(log.get("sample_size", 0) for log in logging_outputs) + agg_output = { + "loss": loss_sum / nsentences, + "ntokens": ntokens, + "nsentences": nsentences, + "sample_size": sample_size, + } + return agg_output diff --git a/data/fairseq/examples/speech_recognition/criterions/__init__.py b/data/fairseq/examples/speech_recognition/criterions/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..579abd2ace1b14b80f5e53e5c96583e4d5b14c52 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/criterions/__init__.py @@ -0,0 +1,17 @@ +import importlib +import os + + +# ASG loss requires flashlight bindings +files_to_skip = set() +try: + import flashlight.lib.sequence.criterion +except ImportError: + files_to_skip.add("ASG_loss.py") + +for file in sorted(os.listdir(os.path.dirname(__file__))): + if file.endswith(".py") and not file.startswith("_") and file not in files_to_skip: + criterion_name = file[: file.find(".py")] + importlib.import_module( + "examples.speech_recognition.criterions." + criterion_name + ) diff --git a/data/fairseq/examples/speech_recognition/criterions/cross_entropy_acc.py b/data/fairseq/examples/speech_recognition/criterions/cross_entropy_acc.py new file mode 100644 index 0000000000000000000000000000000000000000..7c4d8ba3802a2da9467c42b0aa18653c7bbb2ec9 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/criterions/cross_entropy_acc.py @@ -0,0 +1,130 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +from __future__ import absolute_import, division, print_function, unicode_literals + +import logging +import math + +import torch +import torch.nn.functional as F +from fairseq import utils +from fairseq.criterions import FairseqCriterion, register_criterion + + +@register_criterion("cross_entropy_acc") +class CrossEntropyWithAccCriterion(FairseqCriterion): + def __init__(self, task, sentence_avg): + super().__init__(task) + self.sentence_avg = sentence_avg + + def compute_loss(self, model, net_output, target, reduction, log_probs): + # N, T -> N * T + target = target.view(-1) + lprobs = model.get_normalized_probs(net_output, log_probs=log_probs) + if not hasattr(lprobs, "batch_first"): + logging.warning( + "ERROR: we need to know whether " + "batch first for the net output; " + "you need to set batch_first attribute for the return value of " + "model.get_normalized_probs. Now, we assume this is true, but " + "in the future, we will raise exception instead. " + ) + batch_first = getattr(lprobs, "batch_first", True) + if not batch_first: + lprobs = lprobs.transpose(0, 1) + + # N, T, D -> N * T, D + lprobs = lprobs.view(-1, lprobs.size(-1)) + loss = F.nll_loss( + lprobs, target, ignore_index=self.padding_idx, reduction=reduction + ) + return lprobs, loss + + def get_logging_output(self, sample, target, lprobs, loss): + target = target.view(-1) + mask = target != self.padding_idx + correct = torch.sum( + lprobs.argmax(1).masked_select(mask) == target.masked_select(mask) + ) + total = torch.sum(mask) + sample_size = ( + sample["target"].size(0) if self.sentence_avg else sample["ntokens"] + ) + + logging_output = { + "loss": utils.item(loss.data), # * sample['ntokens'], + "ntokens": sample["ntokens"], + "nsentences": sample["target"].size(0), + "sample_size": sample_size, + "correct": utils.item(correct.data), + "total": utils.item(total.data), + "nframes": torch.sum(sample["net_input"]["src_lengths"]).item(), + } + + return sample_size, logging_output + + def forward(self, model, sample, reduction="sum", log_probs=True): + """Computes the cross entropy with accuracy metric for the given sample. + + This is similar to CrossEntropyCriterion in fairseq, but also + computes accuracy metrics as part of logging + + Args: + logprobs (Torch.tensor) of shape N, T, D i.e. + batchsize, timesteps, dimensions + targets (Torch.tensor) of shape N, T i.e batchsize, timesteps + + Returns: + tuple: With three elements: + 1) the loss + 2) the sample size, which is used as the denominator for the gradient + 3) logging outputs to display while training + + TODO: + * Currently this Criterion will only work with LSTMEncoderModels or + FairseqModels which have decoder, or Models which return TorchTensor + as net_output. + We need to make a change to support all FairseqEncoder models. + """ + net_output = model(**sample["net_input"]) + target = model.get_targets(sample, net_output) + lprobs, loss = self.compute_loss( + model, net_output, target, reduction, log_probs + ) + sample_size, logging_output = self.get_logging_output( + sample, target, lprobs, loss + ) + return loss, sample_size, logging_output + + @staticmethod + def aggregate_logging_outputs(logging_outputs): + """Aggregate logging outputs from data parallel training.""" + correct_sum = sum(log.get("correct", 0) for log in logging_outputs) + total_sum = sum(log.get("total", 0) for log in logging_outputs) + loss_sum = sum(log.get("loss", 0) for log in logging_outputs) + ntokens = sum(log.get("ntokens", 0) for log in logging_outputs) + nsentences = sum(log.get("nsentences", 0) for log in logging_outputs) + sample_size = sum(log.get("sample_size", 0) for log in logging_outputs) + nframes = sum(log.get("nframes", 0) for log in logging_outputs) + agg_output = { + "loss": loss_sum / sample_size / math.log(2) if sample_size > 0 else 0.0, + # if args.sentence_avg, then sample_size is nsentences, then loss + # is per-sentence loss; else sample_size is ntokens, the loss + # becomes per-output token loss + "ntokens": ntokens, + "nsentences": nsentences, + "nframes": nframes, + "sample_size": sample_size, + "acc": correct_sum * 100.0 / total_sum if total_sum > 0 else 0.0, + "correct": correct_sum, + "total": total_sum, + # total is the number of validate tokens + } + if sample_size != ntokens: + agg_output["nll_loss"] = loss_sum / ntokens / math.log(2) + # loss: per output token loss + # nll_loss: per sentence loss + return agg_output diff --git a/data/fairseq/examples/speech_recognition/data/__init__.py b/data/fairseq/examples/speech_recognition/data/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..47bb6e24ddf25aa4fd5bf0fe9672f89099efb9b4 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/data/__init__.py @@ -0,0 +1,11 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +from .asr_dataset import AsrDataset + + +__all__ = [ + "AsrDataset", +] diff --git a/data/fairseq/examples/speech_recognition/data/asr_dataset.py b/data/fairseq/examples/speech_recognition/data/asr_dataset.py new file mode 100644 index 0000000000000000000000000000000000000000..63a6fcac85d73b1fce8e4d044b4209b1b67fa8ce --- /dev/null +++ b/data/fairseq/examples/speech_recognition/data/asr_dataset.py @@ -0,0 +1,122 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import os + +import numpy as np +from fairseq.data import FairseqDataset + +from . import data_utils +from .collaters import Seq2SeqCollater + + +class AsrDataset(FairseqDataset): + """ + A dataset representing speech and corresponding transcription. + + Args: + aud_paths: (List[str]): A list of str with paths to audio files. + aud_durations_ms (List[int]): A list of int containing the durations of + audio files. + tgt (List[torch.LongTensor]): A list of LongTensors containing the indices + of target transcriptions. + tgt_dict (~fairseq.data.Dictionary): target vocabulary. + ids (List[str]): A list of utterance IDs. + speakers (List[str]): A list of speakers corresponding to utterances. + num_mel_bins (int): Number of triangular mel-frequency bins (default: 80) + frame_length (float): Frame length in milliseconds (default: 25.0) + frame_shift (float): Frame shift in milliseconds (default: 10.0) + """ + + def __init__( + self, + aud_paths, + aud_durations_ms, + tgt, + tgt_dict, + ids, + speakers, + num_mel_bins=80, + frame_length=25.0, + frame_shift=10.0, + ): + assert frame_length > 0 + assert frame_shift > 0 + assert all(x > frame_length for x in aud_durations_ms) + self.frame_sizes = [ + int(1 + (d - frame_length) / frame_shift) for d in aud_durations_ms + ] + + assert len(aud_paths) > 0 + assert len(aud_paths) == len(aud_durations_ms) + assert len(aud_paths) == len(tgt) + assert len(aud_paths) == len(ids) + assert len(aud_paths) == len(speakers) + self.aud_paths = aud_paths + self.tgt_dict = tgt_dict + self.tgt = tgt + self.ids = ids + self.speakers = speakers + self.num_mel_bins = num_mel_bins + self.frame_length = frame_length + self.frame_shift = frame_shift + + self.s2s_collater = Seq2SeqCollater( + 0, + 1, + pad_index=self.tgt_dict.pad(), + eos_index=self.tgt_dict.eos(), + move_eos_to_beginning=True, + ) + + def __getitem__(self, index): + import torchaudio + import torchaudio.compliance.kaldi as kaldi + + tgt_item = self.tgt[index] if self.tgt is not None else None + + path = self.aud_paths[index] + if not os.path.exists(path): + raise FileNotFoundError("Audio file not found: {}".format(path)) + sound, sample_rate = torchaudio.load_wav(path) + output = kaldi.fbank( + sound, + num_mel_bins=self.num_mel_bins, + frame_length=self.frame_length, + frame_shift=self.frame_shift, + ) + output_cmvn = data_utils.apply_mv_norm(output) + + return {"id": index, "data": [output_cmvn.detach(), tgt_item]} + + def __len__(self): + return len(self.aud_paths) + + def collater(self, samples): + """Merge a list of samples to form a mini-batch. + + Args: + samples (List[int]): sample indices to collate + + Returns: + dict: a mini-batch suitable for forwarding with a Model + """ + return self.s2s_collater.collate(samples) + + def num_tokens(self, index): + return self.frame_sizes[index] + + def size(self, index): + """Return an example's size as a float or tuple. This value is used when + filtering a dataset with ``--max-positions``.""" + return ( + self.frame_sizes[index], + len(self.tgt[index]) if self.tgt is not None else 0, + ) + + def ordered_indices(self): + """Return an ordered list of indices. Batches will be constructed based + on this order.""" + return np.arange(len(self)) diff --git a/data/fairseq/examples/speech_recognition/data/collaters.py b/data/fairseq/examples/speech_recognition/data/collaters.py new file mode 100644 index 0000000000000000000000000000000000000000..6acfec876b87e5a00bc92083b1181301a2a18e3f --- /dev/null +++ b/data/fairseq/examples/speech_recognition/data/collaters.py @@ -0,0 +1,131 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. +""" + This module contains collection of classes which implement + collate functionalities for various tasks. + + Collaters should know what data to expect for each sample + and they should pack / collate them into batches +""" + + +from __future__ import absolute_import, division, print_function, unicode_literals + +import numpy as np +import torch +from fairseq.data import data_utils as fairseq_data_utils + + +class Seq2SeqCollater(object): + """ + Implements collate function mainly for seq2seq tasks + This expects each sample to contain feature (src_tokens) and + targets. + This collator is also used for aligned training task. + """ + + def __init__( + self, + feature_index=0, + label_index=1, + pad_index=1, + eos_index=2, + move_eos_to_beginning=True, + ): + self.feature_index = feature_index + self.label_index = label_index + self.pad_index = pad_index + self.eos_index = eos_index + self.move_eos_to_beginning = move_eos_to_beginning + + def _collate_frames(self, frames): + """Convert a list of 2d frames into a padded 3d tensor + Args: + frames (list): list of 2d frames of size L[i]*f_dim. Where L[i] is + length of i-th frame and f_dim is static dimension of features + Returns: + 3d tensor of size len(frames)*len_max*f_dim where len_max is max of L[i] + """ + len_max = max(frame.size(0) for frame in frames) + f_dim = frames[0].size(1) + res = frames[0].new(len(frames), len_max, f_dim).fill_(0.0) + + for i, v in enumerate(frames): + res[i, : v.size(0)] = v + + return res + + def collate(self, samples): + """ + utility function to collate samples into batch for speech recognition. + """ + if len(samples) == 0: + return {} + + # parse samples into torch tensors + parsed_samples = [] + for s in samples: + # skip invalid samples + if s["data"][self.feature_index] is None: + continue + source = s["data"][self.feature_index] + if isinstance(source, (np.ndarray, np.generic)): + source = torch.from_numpy(source) + target = s["data"][self.label_index] + if isinstance(target, (np.ndarray, np.generic)): + target = torch.from_numpy(target).long() + elif isinstance(target, list): + target = torch.LongTensor(target) + + parsed_sample = {"id": s["id"], "source": source, "target": target} + parsed_samples.append(parsed_sample) + samples = parsed_samples + + id = torch.LongTensor([s["id"] for s in samples]) + frames = self._collate_frames([s["source"] for s in samples]) + # sort samples by descending number of frames + frames_lengths = torch.LongTensor([s["source"].size(0) for s in samples]) + frames_lengths, sort_order = frames_lengths.sort(descending=True) + id = id.index_select(0, sort_order) + frames = frames.index_select(0, sort_order) + + target = None + target_lengths = None + prev_output_tokens = None + if samples[0].get("target", None) is not None: + ntokens = sum(len(s["target"]) for s in samples) + target = fairseq_data_utils.collate_tokens( + [s["target"] for s in samples], + self.pad_index, + self.eos_index, + left_pad=False, + move_eos_to_beginning=False, + ) + target = target.index_select(0, sort_order) + target_lengths = torch.LongTensor( + [s["target"].size(0) for s in samples] + ).index_select(0, sort_order) + prev_output_tokens = fairseq_data_utils.collate_tokens( + [s["target"] for s in samples], + self.pad_index, + self.eos_index, + left_pad=False, + move_eos_to_beginning=self.move_eos_to_beginning, + ) + prev_output_tokens = prev_output_tokens.index_select(0, sort_order) + else: + ntokens = sum(len(s["source"]) for s in samples) + + batch = { + "id": id, + "ntokens": ntokens, + "net_input": {"src_tokens": frames, "src_lengths": frames_lengths}, + "target": target, + "target_lengths": target_lengths, + "nsentences": len(samples), + } + if prev_output_tokens is not None: + batch["net_input"]["prev_output_tokens"] = prev_output_tokens + return batch diff --git a/data/fairseq/examples/speech_recognition/data/data_utils.py b/data/fairseq/examples/speech_recognition/data/data_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..cc4729e63c8ef551b29617d1169a44c24f509ad0 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/data/data_utils.py @@ -0,0 +1,100 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import torch + + +def calc_mean_invstddev(feature): + if len(feature.size()) != 2: + raise ValueError("We expect the input feature to be 2-D tensor") + mean = feature.mean(0) + var = feature.var(0) + # avoid division by ~zero + eps = 1e-8 + if (var < eps).any(): + return mean, 1.0 / (torch.sqrt(var) + eps) + return mean, 1.0 / torch.sqrt(var) + + +def apply_mv_norm(features): + # If there is less than 2 spectrograms, the variance cannot be computed (is NaN) + # and normalization is not possible, so return the item as it is + if features.size(0) < 2: + return features + mean, invstddev = calc_mean_invstddev(features) + res = (features - mean) * invstddev + return res + + +def lengths_to_encoder_padding_mask(lengths, batch_first=False): + """ + convert lengths (a 1-D Long/Int tensor) to 2-D binary tensor + + Args: + lengths: a (B, )-shaped tensor + + Return: + max_length: maximum length of B sequences + encoder_padding_mask: a (max_length, B) binary mask, where + [t, b] = 0 for t < lengths[b] and 1 otherwise + + TODO: + kernelize this function if benchmarking shows this function is slow + """ + max_lengths = torch.max(lengths).item() + bsz = lengths.size(0) + encoder_padding_mask = torch.arange( + max_lengths + ).to( # a (T, ) tensor with [0, ..., T-1] + lengths.device + ).view( # move to the right device + 1, max_lengths + ).expand( # reshape to (1, T)-shaped tensor + bsz, -1 + ) >= lengths.view( # expand to (B, T)-shaped tensor + bsz, 1 + ).expand( + -1, max_lengths + ) + if not batch_first: + return encoder_padding_mask.t(), max_lengths + else: + return encoder_padding_mask, max_lengths + + +def encoder_padding_mask_to_lengths( + encoder_padding_mask, max_lengths, batch_size, device +): + """ + convert encoder_padding_mask (2-D binary tensor) to a 1-D tensor + + Conventionally, encoder output contains a encoder_padding_mask, which is + a 2-D mask in a shape (T, B), whose (t, b) element indicate whether + encoder_out[t, b] is a valid output (=0) or not (=1). Occasionally, we + need to convert this mask tensor to a 1-D tensor in shape (B, ), where + [b] denotes the valid length of b-th sequence + + Args: + encoder_padding_mask: a (T, B)-shaped binary tensor or None; if None, + indicating all are valid + Return: + seq_lengths: a (B,)-shaped tensor, where its (b, )-th element is the + number of valid elements of b-th sequence + + max_lengths: maximum length of all sequence, if encoder_padding_mask is + not None, max_lengths must equal to encoder_padding_mask.size(0) + + batch_size: batch size; if encoder_padding_mask is + not None, max_lengths must equal to encoder_padding_mask.size(1) + + device: which device to put the result on + """ + if encoder_padding_mask is None: + return torch.Tensor([max_lengths] * batch_size).to(torch.int32).to(device) + + assert encoder_padding_mask.size(0) == max_lengths, "max_lengths does not match" + assert encoder_padding_mask.size(1) == batch_size, "batch_size does not match" + + return max_lengths - torch.sum(encoder_padding_mask, dim=0) diff --git a/data/fairseq/examples/speech_recognition/data/replabels.py b/data/fairseq/examples/speech_recognition/data/replabels.py new file mode 100644 index 0000000000000000000000000000000000000000..441f1bd432b95865fc981c6c695cee299b07ed62 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/data/replabels.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 + +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +""" +Replabel transforms for use with flashlight's ASG criterion. +""" + + +def replabel_symbol(i): + """ + Replabel symbols used in flashlight, currently just "1", "2", ... + This prevents training with numeral tokens, so this might change in the future + """ + return str(i) + + +def pack_replabels(tokens, dictionary, max_reps): + """ + Pack a token sequence so that repeated symbols are replaced by replabels + """ + if len(tokens) == 0 or max_reps <= 0: + return tokens + + replabel_value_to_idx = [0] * (max_reps + 1) + for i in range(1, max_reps + 1): + replabel_value_to_idx[i] = dictionary.index(replabel_symbol(i)) + + result = [] + prev_token = -1 + num_reps = 0 + for token in tokens: + if token == prev_token and num_reps < max_reps: + num_reps += 1 + else: + if num_reps > 0: + result.append(replabel_value_to_idx[num_reps]) + num_reps = 0 + result.append(token) + prev_token = token + if num_reps > 0: + result.append(replabel_value_to_idx[num_reps]) + return result + + +def unpack_replabels(tokens, dictionary, max_reps): + """ + Unpack a token sequence so that replabels are replaced by repeated symbols + """ + if len(tokens) == 0 or max_reps <= 0: + return tokens + + replabel_idx_to_value = {} + for i in range(1, max_reps + 1): + replabel_idx_to_value[dictionary.index(replabel_symbol(i))] = i + + result = [] + prev_token = -1 + for token in tokens: + try: + for _ in range(replabel_idx_to_value[token]): + result.append(prev_token) + prev_token = -1 + except KeyError: + result.append(token) + prev_token = token + return result diff --git a/data/fairseq/examples/speech_recognition/datasets/asr_prep_json.py b/data/fairseq/examples/speech_recognition/datasets/asr_prep_json.py new file mode 100644 index 0000000000000000000000000000000000000000..b8db8ff16691158fae034a8ab3faad622b351caf --- /dev/null +++ b/data/fairseq/examples/speech_recognition/datasets/asr_prep_json.py @@ -0,0 +1,125 @@ +#!/usr/bin/env python3 +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +from __future__ import absolute_import, division, print_function, unicode_literals + +import argparse +import concurrent.futures +import json +import multiprocessing +import os +from collections import namedtuple +from itertools import chain + +import sentencepiece as spm +from fairseq.data import Dictionary + + +MILLISECONDS_TO_SECONDS = 0.001 + + +def process_sample(aud_path, lable, utt_id, sp, tgt_dict): + import torchaudio + + input = {} + output = {} + si, ei = torchaudio.info(aud_path) + input["length_ms"] = int( + si.length / si.channels / si.rate / MILLISECONDS_TO_SECONDS + ) + input["path"] = aud_path + + token = " ".join(sp.EncodeAsPieces(lable)) + ids = tgt_dict.encode_line(token, append_eos=False) + output["text"] = lable + output["token"] = token + output["tokenid"] = ", ".join(map(str, [t.tolist() for t in ids])) + return {utt_id: {"input": input, "output": output}} + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument( + "--audio-dirs", + nargs="+", + default=["-"], + required=True, + help="input directories with audio files", + ) + parser.add_argument( + "--labels", + required=True, + help="aggregated input labels with format per line", + type=argparse.FileType("r", encoding="UTF-8"), + ) + parser.add_argument( + "--spm-model", + required=True, + help="sentencepiece model to use for encoding", + type=argparse.FileType("r", encoding="UTF-8"), + ) + parser.add_argument( + "--dictionary", + required=True, + help="file to load fairseq dictionary from", + type=argparse.FileType("r", encoding="UTF-8"), + ) + parser.add_argument("--audio-format", choices=["flac", "wav"], default="wav") + parser.add_argument( + "--output", + required=True, + type=argparse.FileType("w"), + help="path to save json output", + ) + args = parser.parse_args() + + sp = spm.SentencePieceProcessor() + sp.Load(args.spm_model.name) + + tgt_dict = Dictionary.load(args.dictionary) + + labels = {} + for line in args.labels: + (utt_id, label) = line.split(" ", 1) + labels[utt_id] = label + if len(labels) == 0: + raise Exception("No labels found in ", args.labels_path) + + Sample = namedtuple("Sample", "aud_path utt_id") + samples = [] + for path, _, files in chain.from_iterable( + os.walk(path) for path in args.audio_dirs + ): + for f in files: + if f.endswith(args.audio_format): + if len(os.path.splitext(f)) != 2: + raise Exception("Expect file name. Got: ", f) + utt_id = os.path.splitext(f)[0] + if utt_id not in labels: + continue + samples.append(Sample(os.path.join(path, f), utt_id)) + + utts = {} + num_cpu = multiprocessing.cpu_count() + with concurrent.futures.ThreadPoolExecutor(max_workers=num_cpu) as executor: + future_to_sample = { + executor.submit( + process_sample, s.aud_path, labels[s.utt_id], s.utt_id, sp, tgt_dict + ): s + for s in samples + } + for future in concurrent.futures.as_completed(future_to_sample): + try: + data = future.result() + except Exception as exc: + print("generated an exception: ", exc) + else: + utts.update(data) + json.dump({"utts": utts}, args.output, indent=4) + + +if __name__ == "__main__": + main() diff --git a/data/fairseq/examples/speech_recognition/datasets/prepare-librispeech.sh b/data/fairseq/examples/speech_recognition/datasets/prepare-librispeech.sh new file mode 100644 index 0000000000000000000000000000000000000000..9e9297f08947027685ff508bfa91ff26b0d8ea0c --- /dev/null +++ b/data/fairseq/examples/speech_recognition/datasets/prepare-librispeech.sh @@ -0,0 +1,88 @@ +#!/usr/bin/env bash +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +# Prepare librispeech dataset + +base_url=www.openslr.org/resources/12 +train_dir=train_960 + +if [ "$#" -ne 2 ]; then + echo "Usage: $0 " + echo "e.g.: $0 /tmp/librispeech_raw/ ~/data/librispeech_final" + exit 1 +fi + +download_dir=${1%/} +out_dir=${2%/} + +fairseq_root=~/fairseq-py/ +mkdir -p ${out_dir} +cd ${out_dir} || exit + +nbpe=5000 +bpemode=unigram + +if [ ! -d "$fairseq_root" ]; then + echo "$0: Please set correct fairseq_root" + exit 1 +fi + +echo "Data Download" +for part in dev-clean test-clean dev-other test-other train-clean-100 train-clean-360 train-other-500; do + url=$base_url/$part.tar.gz + if ! wget -P $download_dir $url; then + echo "$0: wget failed for $url" + exit 1 + fi + if ! tar -C $download_dir -xvzf $download_dir/$part.tar.gz; then + echo "$0: error un-tarring archive $download_dir/$part.tar.gz" + exit 1 + fi +done + +echo "Merge all train packs into one" +mkdir -p ${download_dir}/LibriSpeech/${train_dir}/ +for part in train-clean-100 train-clean-360 train-other-500; do + mv ${download_dir}/LibriSpeech/${part}/* $download_dir/LibriSpeech/${train_dir}/ +done +echo "Merge train text" +find ${download_dir}/LibriSpeech/${train_dir}/ -name '*.txt' -exec cat {} \; >> ${download_dir}/LibriSpeech/${train_dir}/text + +# Use combined dev-clean and dev-other as validation set +find ${download_dir}/LibriSpeech/dev-clean/ ${download_dir}/LibriSpeech/dev-other/ -name '*.txt' -exec cat {} \; >> ${download_dir}/LibriSpeech/valid_text +find ${download_dir}/LibriSpeech/test-clean/ -name '*.txt' -exec cat {} \; >> ${download_dir}/LibriSpeech/test-clean/text +find ${download_dir}/LibriSpeech/test-other/ -name '*.txt' -exec cat {} \; >> ${download_dir}/LibriSpeech/test-other/text + + +dict=data/lang_char/${train_dir}_${bpemode}${nbpe}_units.txt +encoded=data/lang_char/${train_dir}_${bpemode}${nbpe}_encoded.txt +fairseq_dict=data/lang_char/${train_dir}_${bpemode}${nbpe}_fairseq_dict.txt +bpemodel=data/lang_char/${train_dir}_${bpemode}${nbpe} +echo "dictionary: ${dict}" +echo "Dictionary preparation" +mkdir -p data/lang_char/ +echo " 3" > ${dict} +echo " 2" >> ${dict} +echo " 1" >> ${dict} +cut -f 2- -d" " ${download_dir}/LibriSpeech/${train_dir}/text > data/lang_char/input.txt +spm_train --input=data/lang_char/input.txt --vocab_size=${nbpe} --model_type=${bpemode} --model_prefix=${bpemodel} --input_sentence_size=100000000 --unk_id=3 --eos_id=2 --pad_id=1 --bos_id=-1 --character_coverage=1 +spm_encode --model=${bpemodel}.model --output_format=piece < data/lang_char/input.txt > ${encoded} +cat ${encoded} | tr ' ' '\n' | sort | uniq | awk '{print $0 " " NR+3}' >> ${dict} +cat ${encoded} | tr ' ' '\n' | sort | uniq -c | awk '{print $2 " " $1}' > ${fairseq_dict} +wc -l ${dict} + +echo "Prepare train and test jsons" +for part in train_960 test-other test-clean; do + python ${fairseq_root}/examples/speech_recognition/datasets/asr_prep_json.py --audio-dirs ${download_dir}/LibriSpeech/${part} --labels ${download_dir}/LibriSpeech/${part}/text --spm-model ${bpemodel}.model --audio-format flac --dictionary ${fairseq_dict} --output ${part}.json +done +# fairseq expects to find train.json and valid.json during training +mv train_960.json train.json + +echo "Prepare valid json" +python ${fairseq_root}/examples/speech_recognition/datasets/asr_prep_json.py --audio-dirs ${download_dir}/LibriSpeech/dev-clean ${download_dir}/LibriSpeech/dev-other --labels ${download_dir}/LibriSpeech/valid_text --spm-model ${bpemodel}.model --audio-format flac --dictionary ${fairseq_dict} --output valid.json + +cp ${fairseq_dict} ./dict.txt +cp ${bpemodel}.model ./spm.model diff --git a/data/fairseq/examples/speech_recognition/infer.py b/data/fairseq/examples/speech_recognition/infer.py new file mode 100644 index 0000000000000000000000000000000000000000..ce16bf47cfb63c9de787c36d4a5edd24cde6e421 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/infer.py @@ -0,0 +1,436 @@ +#!/usr/bin/env python3 -u +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +""" +Run inference for pre-processed data with a trained model. +""" + +import ast +import logging +import math +import os +import sys + +import editdistance +import numpy as np +import torch +from fairseq import checkpoint_utils, options, progress_bar, tasks, utils +from fairseq.data.data_utils import post_process +from fairseq.logging.meters import StopwatchMeter, TimeMeter + + +logging.basicConfig() +logging.root.setLevel(logging.INFO) +logging.basicConfig(level=logging.INFO) +logger = logging.getLogger(__name__) + + +def add_asr_eval_argument(parser): + parser.add_argument("--kspmodel", default=None, help="sentence piece model") + parser.add_argument( + "--wfstlm", default=None, help="wfstlm on dictonary output units" + ) + parser.add_argument( + "--rnnt_decoding_type", + default="greedy", + help="wfstlm on dictonary\ +output units", + ) + try: + parser.add_argument( + "--lm-weight", + "--lm_weight", + type=float, + default=0.2, + help="weight for lm while interpolating with neural score", + ) + except: + pass + parser.add_argument( + "--rnnt_len_penalty", default=-0.5, help="rnnt length penalty on word level" + ) + parser.add_argument( + "--w2l-decoder", + choices=["viterbi", "kenlm", "fairseqlm"], + help="use a w2l decoder", + ) + parser.add_argument("--lexicon", help="lexicon for w2l decoder") + parser.add_argument("--unit-lm", action="store_true", help="if using a unit lm") + parser.add_argument("--kenlm-model", "--lm-model", help="lm model for w2l decoder") + parser.add_argument("--beam-threshold", type=float, default=25.0) + parser.add_argument("--beam-size-token", type=float, default=100) + parser.add_argument("--word-score", type=float, default=1.0) + parser.add_argument("--unk-weight", type=float, default=-math.inf) + parser.add_argument("--sil-weight", type=float, default=0.0) + parser.add_argument( + "--dump-emissions", + type=str, + default=None, + help="if present, dumps emissions into this file and exits", + ) + parser.add_argument( + "--dump-features", + type=str, + default=None, + help="if present, dumps features into this file and exits", + ) + parser.add_argument( + "--load-emissions", + type=str, + default=None, + help="if present, loads emissions from this file", + ) + return parser + + +def check_args(args): + # assert args.path is not None, "--path required for generation!" + # assert args.results_path is not None, "--results_path required for generation!" + assert ( + not args.sampling or args.nbest == args.beam + ), "--sampling requires --nbest to be equal to --beam" + assert ( + args.replace_unk is None or args.raw_text + ), "--replace-unk requires a raw text dataset (--raw-text)" + + +def get_dataset_itr(args, task, models): + return task.get_batch_iterator( + dataset=task.dataset(args.gen_subset), + max_tokens=args.max_tokens, + max_sentences=args.batch_size, + max_positions=(sys.maxsize, sys.maxsize), + ignore_invalid_inputs=args.skip_invalid_size_inputs_valid_test, + required_batch_size_multiple=args.required_batch_size_multiple, + num_shards=args.num_shards, + shard_id=args.shard_id, + num_workers=args.num_workers, + data_buffer_size=args.data_buffer_size, + ).next_epoch_itr(shuffle=False) + + +def process_predictions( + args, hypos, sp, tgt_dict, target_tokens, res_files, speaker, id +): + for hypo in hypos[: min(len(hypos), args.nbest)]: + hyp_pieces = tgt_dict.string(hypo["tokens"].int().cpu()) + + if "words" in hypo: + hyp_words = " ".join(hypo["words"]) + else: + hyp_words = post_process(hyp_pieces, args.post_process) + + if res_files is not None: + print( + "{} ({}-{})".format(hyp_pieces, speaker, id), + file=res_files["hypo.units"], + ) + print( + "{} ({}-{})".format(hyp_words, speaker, id), + file=res_files["hypo.words"], + ) + + tgt_pieces = tgt_dict.string(target_tokens) + tgt_words = post_process(tgt_pieces, args.post_process) + + if res_files is not None: + print( + "{} ({}-{})".format(tgt_pieces, speaker, id), + file=res_files["ref.units"], + ) + print( + "{} ({}-{})".format(tgt_words, speaker, id), file=res_files["ref.words"] + ) + + if not args.quiet: + logger.info("HYPO:" + hyp_words) + logger.info("TARGET:" + tgt_words) + logger.info("___________________") + + hyp_words = hyp_words.split() + tgt_words = tgt_words.split() + return editdistance.eval(hyp_words, tgt_words), len(tgt_words) + + +def prepare_result_files(args): + def get_res_file(file_prefix): + if args.num_shards > 1: + file_prefix = f"{args.shard_id}_{file_prefix}" + path = os.path.join( + args.results_path, + "{}-{}-{}.txt".format( + file_prefix, os.path.basename(args.path), args.gen_subset + ), + ) + return open(path, "w", buffering=1) + + if not args.results_path: + return None + + return { + "hypo.words": get_res_file("hypo.word"), + "hypo.units": get_res_file("hypo.units"), + "ref.words": get_res_file("ref.word"), + "ref.units": get_res_file("ref.units"), + } + + +def optimize_models(args, use_cuda, models): + """Optimize ensemble for generation""" + for model in models: + model.make_generation_fast_( + beamable_mm_beam_size=None if args.no_beamable_mm else args.beam, + need_attn=args.print_alignment, + ) + if args.fp16: + model.half() + if use_cuda: + model.cuda() + + +def apply_half(t): + if t.dtype is torch.float32: + return t.to(dtype=torch.half) + return t + + +class ExistingEmissionsDecoder(object): + def __init__(self, decoder, emissions): + self.decoder = decoder + self.emissions = emissions + + def generate(self, models, sample, **unused): + ids = sample["id"].cpu().numpy() + try: + emissions = np.stack(self.emissions[ids]) + except: + print([x.shape for x in self.emissions[ids]]) + raise Exception("invalid sizes") + emissions = torch.from_numpy(emissions) + return self.decoder.decode(emissions) + + +def main(args, task=None, model_state=None): + check_args(args) + + use_fp16 = args.fp16 + if args.max_tokens is None and args.batch_size is None: + args.max_tokens = 4000000 + logger.info(args) + + use_cuda = torch.cuda.is_available() and not args.cpu + + logger.info("| decoding with criterion {}".format(args.criterion)) + + task = tasks.setup_task(args) + + # Load ensemble + if args.load_emissions: + models, criterions = [], [] + task.load_dataset(args.gen_subset) + else: + logger.info("| loading model(s) from {}".format(args.path)) + models, saved_cfg, task = checkpoint_utils.load_model_ensemble_and_task( + utils.split_paths(args.path, separator="\\"), + arg_overrides=ast.literal_eval(args.model_overrides), + task=task, + suffix=args.checkpoint_suffix, + strict=(args.checkpoint_shard_count == 1), + num_shards=args.checkpoint_shard_count, + state=model_state, + ) + optimize_models(args, use_cuda, models) + task.load_dataset(args.gen_subset, task_cfg=saved_cfg.task) + + + # Set dictionary + tgt_dict = task.target_dictionary + + logger.info( + "| {} {} {} examples".format( + args.data, args.gen_subset, len(task.dataset(args.gen_subset)) + ) + ) + + # hack to pass transitions to W2lDecoder + if args.criterion == "asg_loss": + raise NotImplementedError("asg_loss is currently not supported") + # trans = criterions[0].asg.trans.data + # args.asg_transitions = torch.flatten(trans).tolist() + + # Load dataset (possibly sharded) + itr = get_dataset_itr(args, task, models) + + # Initialize generator + gen_timer = StopwatchMeter() + + def build_generator(args): + w2l_decoder = getattr(args, "w2l_decoder", None) + if w2l_decoder == "viterbi": + from examples.speech_recognition.w2l_decoder import W2lViterbiDecoder + + return W2lViterbiDecoder(args, task.target_dictionary) + elif w2l_decoder == "kenlm": + from examples.speech_recognition.w2l_decoder import W2lKenLMDecoder + + return W2lKenLMDecoder(args, task.target_dictionary) + elif w2l_decoder == "fairseqlm": + from examples.speech_recognition.w2l_decoder import W2lFairseqLMDecoder + + return W2lFairseqLMDecoder(args, task.target_dictionary) + else: + print( + "only flashlight decoders with (viterbi, kenlm, fairseqlm) options are supported at the moment" + ) + + # please do not touch this unless you test both generate.py and infer.py with audio_pretraining task + generator = build_generator(args) + + if args.load_emissions: + generator = ExistingEmissionsDecoder( + generator, np.load(args.load_emissions, allow_pickle=True) + ) + logger.info("loaded emissions from " + args.load_emissions) + + num_sentences = 0 + + if args.results_path is not None and not os.path.exists(args.results_path): + os.makedirs(args.results_path) + + max_source_pos = ( + utils.resolve_max_positions( + task.max_positions(), *[model.max_positions() for model in models] + ), + ) + + if max_source_pos is not None: + max_source_pos = max_source_pos[0] + if max_source_pos is not None: + max_source_pos = max_source_pos[0] - 1 + + if args.dump_emissions: + emissions = {} + if args.dump_features: + features = {} + models[0].bert.proj = None + else: + res_files = prepare_result_files(args) + errs_t = 0 + lengths_t = 0 + with progress_bar.build_progress_bar(args, itr) as t: + wps_meter = TimeMeter() + for sample in t: + sample = utils.move_to_cuda(sample) if use_cuda else sample + if use_fp16: + sample = utils.apply_to_sample(apply_half, sample) + if "net_input" not in sample: + continue + + prefix_tokens = None + if args.prefix_size > 0: + prefix_tokens = sample["target"][:, : args.prefix_size] + + gen_timer.start() + if args.dump_emissions: + with torch.no_grad(): + encoder_out = models[0](**sample["net_input"]) + emm = models[0].get_normalized_probs(encoder_out, log_probs=True) + emm = emm.transpose(0, 1).cpu().numpy() + for i, id in enumerate(sample["id"]): + emissions[id.item()] = emm[i] + continue + elif args.dump_features: + with torch.no_grad(): + encoder_out = models[0](**sample["net_input"]) + feat = encoder_out["encoder_out"].transpose(0, 1).cpu().numpy() + for i, id in enumerate(sample["id"]): + padding = ( + encoder_out["encoder_padding_mask"][i].cpu().numpy() + if encoder_out["encoder_padding_mask"] is not None + else None + ) + features[id.item()] = (feat[i], padding) + continue + hypos = task.inference_step(generator, models, sample, prefix_tokens) + num_generated_tokens = sum(len(h[0]["tokens"]) for h in hypos) + gen_timer.stop(num_generated_tokens) + + for i, sample_id in enumerate(sample["id"].tolist()): + speaker = None + # id = task.dataset(args.gen_subset).ids[int(sample_id)] + id = sample_id + toks = ( + sample["target"][i, :] + if "target_label" not in sample + else sample["target_label"][i, :] + ) + target_tokens = utils.strip_pad(toks, tgt_dict.pad()).int().cpu() + # Process top predictions + errs, length = process_predictions( + args, + hypos[i], + None, + tgt_dict, + target_tokens, + res_files, + speaker, + id, + ) + errs_t += errs + lengths_t += length + + wps_meter.update(num_generated_tokens) + t.log({"wps": round(wps_meter.avg)}) + num_sentences += ( + sample["nsentences"] if "nsentences" in sample else sample["id"].numel() + ) + + wer = None + if args.dump_emissions: + emm_arr = [] + for i in range(len(emissions)): + emm_arr.append(emissions[i]) + np.save(args.dump_emissions, emm_arr) + logger.info(f"saved {len(emissions)} emissions to {args.dump_emissions}") + elif args.dump_features: + feat_arr = [] + for i in range(len(features)): + feat_arr.append(features[i]) + np.save(args.dump_features, feat_arr) + logger.info(f"saved {len(features)} emissions to {args.dump_features}") + else: + if lengths_t > 0: + wer = errs_t * 100.0 / lengths_t + logger.info(f"WER: {wer}") + + logger.info( + "| Processed {} sentences ({} tokens) in {:.1f}s ({:.2f}" + "sentences/s, {:.2f} tokens/s)".format( + num_sentences, + gen_timer.n, + gen_timer.sum, + num_sentences / gen_timer.sum, + 1.0 / gen_timer.avg, + ) + ) + logger.info("| Generate {} with beam={}".format(args.gen_subset, args.beam)) + return task, wer + + +def make_parser(): + parser = options.get_generation_parser() + parser = add_asr_eval_argument(parser) + return parser + + +def cli_main(): + parser = make_parser() + args = options.parse_args_and_arch(parser) + main(args) + + +if __name__ == "__main__": + cli_main() diff --git a/data/fairseq/examples/speech_recognition/kaldi/__init__.py b/data/fairseq/examples/speech_recognition/kaldi/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/data/fairseq/examples/speech_recognition/kaldi/add-self-loop-simple.cc b/data/fairseq/examples/speech_recognition/kaldi/add-self-loop-simple.cc new file mode 100644 index 0000000000000000000000000000000000000000..e18fb62df52ab85d7802615d8619b0fd94a08f8c --- /dev/null +++ b/data/fairseq/examples/speech_recognition/kaldi/add-self-loop-simple.cc @@ -0,0 +1,94 @@ +/* + * Copyright (c) Facebook, Inc. and its affiliates. + * + * This source code is licensed under the MIT license found in the + * LICENSE file in the root directory of this source tree. + */ + +#include +#include "fstext/fstext-lib.h" // @manual +#include "util/common-utils.h" // @manual + +/* + * This program is to modify a FST without self-loop by: + * for each incoming arc with non-eps input symbol, add a self-loop arc + * with that non-eps symbol as input and eps as output. + * + * This is to make sure the resultant FST can do deduplication for repeated + * symbols, which is very common in acoustic model + * + */ +namespace { +int32 AddSelfLoopsSimple(fst::StdVectorFst* fst) { + typedef fst::MutableArcIterator IterType; + + int32 num_states_before = fst->NumStates(); + fst::MakePrecedingInputSymbolsSame(false, fst); + int32 num_states_after = fst->NumStates(); + KALDI_LOG << "There are " << num_states_before + << " states in the original FST; " + << " after MakePrecedingInputSymbolsSame, there are " + << num_states_after << " states " << std::endl; + + auto weight_one = fst::StdArc::Weight::One(); + + int32 num_arc_added = 0; + + fst::StdArc self_loop_arc; + self_loop_arc.weight = weight_one; + + int32 num_states = fst->NumStates(); + std::vector> incoming_non_eps_label_per_state(num_states); + + for (int32 state = 0; state < num_states; state++) { + for (IterType aiter(fst, state); !aiter.Done(); aiter.Next()) { + fst::StdArc arc(aiter.Value()); + if (arc.ilabel != 0) { + incoming_non_eps_label_per_state[arc.nextstate].insert(arc.ilabel); + } + } + } + + for (int32 state = 0; state < num_states; state++) { + if (!incoming_non_eps_label_per_state[state].empty()) { + auto& ilabel_set = incoming_non_eps_label_per_state[state]; + for (auto it = ilabel_set.begin(); it != ilabel_set.end(); it++) { + self_loop_arc.ilabel = *it; + self_loop_arc.olabel = 0; + self_loop_arc.nextstate = state; + fst->AddArc(state, self_loop_arc); + num_arc_added++; + } + } + } + return num_arc_added; +} + +void print_usage() { + std::cout << "add-self-loop-simple usage:\n" + "\tadd-self-loop-simple \n"; +} +} // namespace + +int main(int argc, char** argv) { + if (argc != 3) { + print_usage(); + exit(1); + } + + auto input = argv[1]; + auto output = argv[2]; + + auto fst = fst::ReadFstKaldi(input); + auto num_states = fst->NumStates(); + KALDI_LOG << "Loading FST from " << input << " with " << num_states + << " states." << std::endl; + + int32 num_arc_added = AddSelfLoopsSimple(fst); + KALDI_LOG << "Adding " << num_arc_added << " self-loop arcs " << std::endl; + + fst::WriteFstKaldi(*fst, std::string(output)); + KALDI_LOG << "Writing FST to " << output << std::endl; + + delete fst; +} diff --git a/data/fairseq/examples/speech_recognition/kaldi/config/kaldi_initializer.yaml b/data/fairseq/examples/speech_recognition/kaldi/config/kaldi_initializer.yaml new file mode 100644 index 0000000000000000000000000000000000000000..be9ba98f55463d41d5d5ea35e306abc0886dbead --- /dev/null +++ b/data/fairseq/examples/speech_recognition/kaldi/config/kaldi_initializer.yaml @@ -0,0 +1,8 @@ +# @package _group_ + +data_dir: ??? +fst_dir: ??? +in_labels: ??? +kaldi_root: ??? +lm_arpa: ??? +blank_symbol: diff --git a/data/fairseq/examples/speech_recognition/kaldi/kaldi_decoder.py b/data/fairseq/examples/speech_recognition/kaldi/kaldi_decoder.py new file mode 100644 index 0000000000000000000000000000000000000000..5f62cc58ae8c0c5a3ba7d17713fedf0abc302942 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/kaldi/kaldi_decoder.py @@ -0,0 +1,244 @@ +#!/usr/bin/env python3 + +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +from concurrent.futures import ThreadPoolExecutor +import logging +from omegaconf import MISSING +import os +import torch +from typing import Optional +import warnings + + +from dataclasses import dataclass +from fairseq.dataclass import FairseqDataclass +from .kaldi_initializer import KaldiInitializerConfig, initalize_kaldi + + +logger = logging.getLogger(__name__) + + +@dataclass +class KaldiDecoderConfig(FairseqDataclass): + hlg_graph_path: Optional[str] = None + output_dict: str = MISSING + + kaldi_initializer_config: Optional[KaldiInitializerConfig] = None + + acoustic_scale: float = 0.5 + max_active: int = 10000 + beam_delta: float = 0.5 + hash_ratio: float = 2.0 + + is_lattice: bool = False + lattice_beam: float = 10.0 + prune_interval: int = 25 + determinize_lattice: bool = True + prune_scale: float = 0.1 + max_mem: int = 0 + phone_determinize: bool = True + word_determinize: bool = True + minimize: bool = True + + num_threads: int = 1 + + +class KaldiDecoder(object): + def __init__( + self, + cfg: KaldiDecoderConfig, + beam: int, + nbest: int = 1, + ): + try: + from kaldi.asr import FasterRecognizer, LatticeFasterRecognizer + from kaldi.base import set_verbose_level + from kaldi.decoder import ( + FasterDecoder, + FasterDecoderOptions, + LatticeFasterDecoder, + LatticeFasterDecoderOptions, + ) + from kaldi.lat.functions import DeterminizeLatticePhonePrunedOptions + from kaldi.fstext import read_fst_kaldi, SymbolTable + except: + warnings.warn( + "pykaldi is required for this functionality. Please install from https://github.com/pykaldi/pykaldi" + ) + + # set_verbose_level(2) + + self.acoustic_scale = cfg.acoustic_scale + self.nbest = nbest + + if cfg.hlg_graph_path is None: + assert ( + cfg.kaldi_initializer_config is not None + ), "Must provide hlg graph path or kaldi initializer config" + cfg.hlg_graph_path = initalize_kaldi(cfg.kaldi_initializer_config) + + assert os.path.exists(cfg.hlg_graph_path), cfg.hlg_graph_path + + if cfg.is_lattice: + self.dec_cls = LatticeFasterDecoder + opt_cls = LatticeFasterDecoderOptions + self.rec_cls = LatticeFasterRecognizer + else: + assert self.nbest == 1, "nbest > 1 requires lattice decoder" + self.dec_cls = FasterDecoder + opt_cls = FasterDecoderOptions + self.rec_cls = FasterRecognizer + + self.decoder_options = opt_cls() + self.decoder_options.beam = beam + self.decoder_options.max_active = cfg.max_active + self.decoder_options.beam_delta = cfg.beam_delta + self.decoder_options.hash_ratio = cfg.hash_ratio + + if cfg.is_lattice: + self.decoder_options.lattice_beam = cfg.lattice_beam + self.decoder_options.prune_interval = cfg.prune_interval + self.decoder_options.determinize_lattice = cfg.determinize_lattice + self.decoder_options.prune_scale = cfg.prune_scale + det_opts = DeterminizeLatticePhonePrunedOptions() + det_opts.max_mem = cfg.max_mem + det_opts.phone_determinize = cfg.phone_determinize + det_opts.word_determinize = cfg.word_determinize + det_opts.minimize = cfg.minimize + self.decoder_options.det_opts = det_opts + + self.output_symbols = {} + with open(cfg.output_dict, "r") as f: + for line in f: + items = line.rstrip().split() + assert len(items) == 2 + self.output_symbols[int(items[1])] = items[0] + + logger.info(f"Loading FST from {cfg.hlg_graph_path}") + self.fst = read_fst_kaldi(cfg.hlg_graph_path) + self.symbol_table = SymbolTable.read_text(cfg.output_dict) + + self.executor = ThreadPoolExecutor(max_workers=cfg.num_threads) + + def generate(self, models, sample, **unused): + """Generate a batch of inferences.""" + # model.forward normally channels prev_output_tokens into the decoder + # separately, but SequenceGenerator directly calls model.encoder + encoder_input = { + k: v for k, v in sample["net_input"].items() if k != "prev_output_tokens" + } + emissions, padding = self.get_emissions(models, encoder_input) + return self.decode(emissions, padding) + + def get_emissions(self, models, encoder_input): + """Run encoder and normalize emissions""" + model = models[0] + + all_encoder_out = [m(**encoder_input) for m in models] + + if len(all_encoder_out) > 1: + + if "encoder_out" in all_encoder_out[0]: + encoder_out = { + "encoder_out": sum(e["encoder_out"] for e in all_encoder_out) + / len(all_encoder_out), + "encoder_padding_mask": all_encoder_out[0]["encoder_padding_mask"], + } + padding = encoder_out["encoder_padding_mask"] + else: + encoder_out = { + "logits": sum(e["logits"] for e in all_encoder_out) + / len(all_encoder_out), + "padding_mask": all_encoder_out[0]["padding_mask"], + } + padding = encoder_out["padding_mask"] + else: + encoder_out = all_encoder_out[0] + padding = ( + encoder_out["padding_mask"] + if "padding_mask" in encoder_out + else encoder_out["encoder_padding_mask"] + ) + + if hasattr(model, "get_logits"): + emissions = model.get_logits(encoder_out, normalize=True) + else: + emissions = model.get_normalized_probs(encoder_out, log_probs=True) + + return ( + emissions.cpu().float().transpose(0, 1), + padding.cpu() if padding is not None and padding.any() else None, + ) + + def decode_one(self, logits, padding): + from kaldi.matrix import Matrix + + decoder = self.dec_cls(self.fst, self.decoder_options) + asr = self.rec_cls( + decoder, self.symbol_table, acoustic_scale=self.acoustic_scale + ) + + if padding is not None: + logits = logits[~padding] + + mat = Matrix(logits.numpy()) + + out = asr.decode(mat) + + if self.nbest > 1: + from kaldi.fstext import shortestpath + from kaldi.fstext.utils import ( + convert_compact_lattice_to_lattice, + convert_lattice_to_std, + convert_nbest_to_list, + get_linear_symbol_sequence, + ) + + lat = out["lattice"] + + sp = shortestpath(lat, nshortest=self.nbest) + + sp = convert_compact_lattice_to_lattice(sp) + sp = convert_lattice_to_std(sp) + seq = convert_nbest_to_list(sp) + + results = [] + for s in seq: + _, o, w = get_linear_symbol_sequence(s) + words = list(self.output_symbols[z] for z in o) + results.append( + { + "tokens": words, + "words": words, + "score": w.value, + "emissions": logits, + } + ) + return results + else: + words = out["text"].split() + return [ + { + "tokens": words, + "words": words, + "score": out["likelihood"], + "emissions": logits, + } + ] + + def decode(self, emissions, padding): + if padding is None: + padding = [None] * len(emissions) + + ret = list( + map( + lambda e, p: self.executor.submit(self.decode_one, e, p), + emissions, + padding, + ) + ) + return ret diff --git a/data/fairseq/examples/speech_recognition/kaldi/kaldi_initializer.py b/data/fairseq/examples/speech_recognition/kaldi/kaldi_initializer.py new file mode 100644 index 0000000000000000000000000000000000000000..6d2a2a4b6b809ba1106f9a57cb6f241dc083e670 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/kaldi/kaldi_initializer.py @@ -0,0 +1,698 @@ +#!/usr/bin/env python3 + +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +from dataclasses import dataclass +import hydra +from hydra.core.config_store import ConfigStore +import logging +from omegaconf import MISSING, OmegaConf +import os +import os.path as osp +from pathlib import Path +import subprocess +from typing import Optional + +from fairseq.data.dictionary import Dictionary +from fairseq.dataclass import FairseqDataclass + +script_dir = Path(__file__).resolve().parent +config_path = script_dir / "config" + + +logger = logging.getLogger(__name__) + + +@dataclass +class KaldiInitializerConfig(FairseqDataclass): + data_dir: str = MISSING + fst_dir: Optional[str] = None + in_labels: str = MISSING + out_labels: Optional[str] = None + wav2letter_lexicon: Optional[str] = None + lm_arpa: str = MISSING + kaldi_root: str = MISSING + blank_symbol: str = "" + silence_symbol: Optional[str] = None + + +def create_units(fst_dir: Path, in_labels: str, vocab: Dictionary) -> Path: + in_units_file = fst_dir / f"kaldi_dict.{in_labels}.txt" + if not in_units_file.exists(): + + logger.info(f"Creating {in_units_file}") + + with open(in_units_file, "w") as f: + print(" 0", file=f) + i = 1 + for symb in vocab.symbols[vocab.nspecial :]: + if not symb.startswith("madeupword"): + print(f"{symb} {i}", file=f) + i += 1 + return in_units_file + + +def create_lexicon( + cfg: KaldiInitializerConfig, + fst_dir: Path, + unique_label: str, + in_units_file: Path, + out_words_file: Path, +) -> (Path, Path): + + disambig_in_units_file = fst_dir / f"kaldi_dict.{cfg.in_labels}_disambig.txt" + lexicon_file = fst_dir / f"kaldi_lexicon.{unique_label}.txt" + disambig_lexicon_file = fst_dir / f"kaldi_lexicon.{unique_label}_disambig.txt" + if ( + not lexicon_file.exists() + or not disambig_lexicon_file.exists() + or not disambig_in_units_file.exists() + ): + logger.info(f"Creating {lexicon_file} (in units file: {in_units_file})") + + assert cfg.wav2letter_lexicon is not None or cfg.in_labels == cfg.out_labels + + if cfg.wav2letter_lexicon is not None: + lm_words = set() + with open(out_words_file, "r") as lm_dict_f: + for line in lm_dict_f: + lm_words.add(line.split()[0]) + + num_skipped = 0 + total = 0 + with open(cfg.wav2letter_lexicon, "r") as w2l_lex_f, open( + lexicon_file, "w" + ) as out_f: + for line in w2l_lex_f: + items = line.rstrip().split("\t") + assert len(items) == 2, items + if items[0] in lm_words: + print(items[0], items[1], file=out_f) + else: + num_skipped += 1 + logger.debug( + f"Skipping word {items[0]} as it was not found in LM" + ) + total += 1 + if num_skipped > 0: + logger.warning( + f"Skipped {num_skipped} out of {total} words as they were not found in LM" + ) + else: + with open(in_units_file, "r") as in_f, open(lexicon_file, "w") as out_f: + for line in in_f: + symb = line.split()[0] + if symb != "" and symb != "" and symb != "": + print(symb, symb, file=out_f) + + lex_disambig_path = ( + Path(cfg.kaldi_root) / "egs/wsj/s5/utils/add_lex_disambig.pl" + ) + res = subprocess.run( + [lex_disambig_path, lexicon_file, disambig_lexicon_file], + check=True, + capture_output=True, + ) + ndisambig = int(res.stdout) + disamib_path = Path(cfg.kaldi_root) / "egs/wsj/s5/utils/add_disambig.pl" + res = subprocess.run( + [disamib_path, "--include-zero", in_units_file, str(ndisambig)], + check=True, + capture_output=True, + ) + with open(disambig_in_units_file, "wb") as f: + f.write(res.stdout) + + return disambig_lexicon_file, disambig_in_units_file + + +def create_G( + kaldi_root: Path, fst_dir: Path, lm_arpa: Path, arpa_base: str +) -> (Path, Path): + + out_words_file = fst_dir / f"kaldi_dict.{arpa_base}.txt" + grammar_graph = fst_dir / f"G_{arpa_base}.fst" + if not grammar_graph.exists() or not out_words_file.exists(): + logger.info(f"Creating {grammar_graph}") + arpa2fst = kaldi_root / "src/lmbin/arpa2fst" + subprocess.run( + [ + arpa2fst, + "--disambig-symbol=#0", + f"--write-symbol-table={out_words_file}", + lm_arpa, + grammar_graph, + ], + check=True, + ) + return grammar_graph, out_words_file + + +def create_L( + kaldi_root: Path, + fst_dir: Path, + unique_label: str, + lexicon_file: Path, + in_units_file: Path, + out_words_file: Path, +) -> Path: + lexicon_graph = fst_dir / f"L.{unique_label}.fst" + + if not lexicon_graph.exists(): + logger.info(f"Creating {lexicon_graph} (in units: {in_units_file})") + make_lex = kaldi_root / "egs/wsj/s5/utils/make_lexicon_fst.pl" + fstcompile = kaldi_root / "tools/openfst-1.6.7/bin/fstcompile" + fstaddselfloops = kaldi_root / "src/fstbin/fstaddselfloops" + fstarcsort = kaldi_root / "tools/openfst-1.6.7/bin/fstarcsort" + + def write_disambig_symbol(file): + with open(file, "r") as f: + for line in f: + items = line.rstrip().split() + if items[0] == "#0": + out_path = str(file) + "_disamig" + with open(out_path, "w") as out_f: + print(items[1], file=out_f) + return out_path + + return None + + in_disambig_sym = write_disambig_symbol(in_units_file) + assert in_disambig_sym is not None + out_disambig_sym = write_disambig_symbol(out_words_file) + assert out_disambig_sym is not None + + try: + with open(lexicon_graph, "wb") as out_f: + res = subprocess.run( + [make_lex, lexicon_file], capture_output=True, check=True + ) + assert len(res.stderr) == 0, res.stderr.decode("utf-8") + res = subprocess.run( + [ + fstcompile, + f"--isymbols={in_units_file}", + f"--osymbols={out_words_file}", + "--keep_isymbols=false", + "--keep_osymbols=false", + ], + input=res.stdout, + capture_output=True, + ) + assert len(res.stderr) == 0, res.stderr.decode("utf-8") + res = subprocess.run( + [fstaddselfloops, in_disambig_sym, out_disambig_sym], + input=res.stdout, + capture_output=True, + check=True, + ) + res = subprocess.run( + [fstarcsort, "--sort_type=olabel"], + input=res.stdout, + capture_output=True, + check=True, + ) + out_f.write(res.stdout) + except subprocess.CalledProcessError as e: + logger.error(f"cmd: {e.cmd}, err: {e.stderr.decode('utf-8')}") + os.remove(lexicon_graph) + raise + except AssertionError: + os.remove(lexicon_graph) + raise + + return lexicon_graph + + +def create_LG( + kaldi_root: Path, + fst_dir: Path, + unique_label: str, + lexicon_graph: Path, + grammar_graph: Path, +) -> Path: + lg_graph = fst_dir / f"LG.{unique_label}.fst" + + if not lg_graph.exists(): + logger.info(f"Creating {lg_graph}") + + fsttablecompose = kaldi_root / "src/fstbin/fsttablecompose" + fstdeterminizestar = kaldi_root / "src/fstbin/fstdeterminizestar" + fstminimizeencoded = kaldi_root / "src/fstbin/fstminimizeencoded" + fstpushspecial = kaldi_root / "src/fstbin/fstpushspecial" + fstarcsort = kaldi_root / "tools/openfst-1.6.7/bin/fstarcsort" + + try: + with open(lg_graph, "wb") as out_f: + res = subprocess.run( + [fsttablecompose, lexicon_graph, grammar_graph], + capture_output=True, + check=True, + ) + res = subprocess.run( + [ + fstdeterminizestar, + "--use-log=true", + ], + input=res.stdout, + capture_output=True, + ) + res = subprocess.run( + [fstminimizeencoded], + input=res.stdout, + capture_output=True, + check=True, + ) + res = subprocess.run( + [fstpushspecial], + input=res.stdout, + capture_output=True, + check=True, + ) + res = subprocess.run( + [fstarcsort, "--sort_type=ilabel"], + input=res.stdout, + capture_output=True, + check=True, + ) + out_f.write(res.stdout) + except subprocess.CalledProcessError as e: + logger.error(f"cmd: {e.cmd}, err: {e.stderr.decode('utf-8')}") + os.remove(lg_graph) + raise + + return lg_graph + + +def create_H( + kaldi_root: Path, + fst_dir: Path, + disambig_out_units_file: Path, + in_labels: str, + vocab: Dictionary, + blk_sym: str, + silence_symbol: Optional[str], +) -> (Path, Path, Path): + h_graph = ( + fst_dir / f"H.{in_labels}{'_' + silence_symbol if silence_symbol else ''}.fst" + ) + h_out_units_file = fst_dir / f"kaldi_dict.h_out.{in_labels}.txt" + disambig_in_units_file_int = Path(str(h_graph) + "isym_disambig.int") + disambig_out_units_file_int = Path(str(disambig_out_units_file) + ".int") + if ( + not h_graph.exists() + or not h_out_units_file.exists() + or not disambig_in_units_file_int.exists() + ): + logger.info(f"Creating {h_graph}") + eps_sym = "" + + num_disambig = 0 + osymbols = [] + + with open(disambig_out_units_file, "r") as f, open( + disambig_out_units_file_int, "w" + ) as out_f: + for line in f: + symb, id = line.rstrip().split() + if line.startswith("#"): + num_disambig += 1 + print(id, file=out_f) + else: + if len(osymbols) == 0: + assert symb == eps_sym, symb + osymbols.append((symb, id)) + + i_idx = 0 + isymbols = [(eps_sym, 0)] + + imap = {} + + for i, s in enumerate(vocab.symbols): + i_idx += 1 + isymbols.append((s, i_idx)) + imap[s] = i_idx + + fst_str = [] + + node_idx = 0 + root_node = node_idx + + special_symbols = [blk_sym] + if silence_symbol is not None: + special_symbols.append(silence_symbol) + + for ss in special_symbols: + fst_str.append("{} {} {} {}".format(root_node, root_node, ss, eps_sym)) + + for symbol, _ in osymbols: + if symbol == eps_sym or symbol.startswith("#"): + continue + + node_idx += 1 + # 1. from root to emitting state + fst_str.append("{} {} {} {}".format(root_node, node_idx, symbol, symbol)) + # 2. from emitting state back to root + fst_str.append("{} {} {} {}".format(node_idx, root_node, eps_sym, eps_sym)) + # 3. from emitting state to optional blank state + pre_node = node_idx + node_idx += 1 + for ss in special_symbols: + fst_str.append("{} {} {} {}".format(pre_node, node_idx, ss, eps_sym)) + # 4. from blank state back to root + fst_str.append("{} {} {} {}".format(node_idx, root_node, eps_sym, eps_sym)) + + fst_str.append("{}".format(root_node)) + + fst_str = "\n".join(fst_str) + h_str = str(h_graph) + isym_file = h_str + ".isym" + + with open(isym_file, "w") as f: + for sym, id in isymbols: + f.write("{} {}\n".format(sym, id)) + + with open(h_out_units_file, "w") as f: + for sym, id in osymbols: + f.write("{} {}\n".format(sym, id)) + + with open(disambig_in_units_file_int, "w") as f: + disam_sym_id = len(isymbols) + for _ in range(num_disambig): + f.write("{}\n".format(disam_sym_id)) + disam_sym_id += 1 + + fstcompile = kaldi_root / "tools/openfst-1.6.7/bin/fstcompile" + fstaddselfloops = kaldi_root / "src/fstbin/fstaddselfloops" + fstarcsort = kaldi_root / "tools/openfst-1.6.7/bin/fstarcsort" + + try: + with open(h_graph, "wb") as out_f: + res = subprocess.run( + [ + fstcompile, + f"--isymbols={isym_file}", + f"--osymbols={h_out_units_file}", + "--keep_isymbols=false", + "--keep_osymbols=false", + ], + input=str.encode(fst_str), + capture_output=True, + check=True, + ) + res = subprocess.run( + [ + fstaddselfloops, + disambig_in_units_file_int, + disambig_out_units_file_int, + ], + input=res.stdout, + capture_output=True, + check=True, + ) + res = subprocess.run( + [fstarcsort, "--sort_type=olabel"], + input=res.stdout, + capture_output=True, + check=True, + ) + out_f.write(res.stdout) + except subprocess.CalledProcessError as e: + logger.error(f"cmd: {e.cmd}, err: {e.stderr.decode('utf-8')}") + os.remove(h_graph) + raise + return h_graph, h_out_units_file, disambig_in_units_file_int + + +def create_HLGa( + kaldi_root: Path, + fst_dir: Path, + unique_label: str, + h_graph: Path, + lg_graph: Path, + disambig_in_words_file_int: Path, +) -> Path: + hlga_graph = fst_dir / f"HLGa.{unique_label}.fst" + + if not hlga_graph.exists(): + logger.info(f"Creating {hlga_graph}") + + fsttablecompose = kaldi_root / "src/fstbin/fsttablecompose" + fstdeterminizestar = kaldi_root / "src/fstbin/fstdeterminizestar" + fstrmsymbols = kaldi_root / "src/fstbin/fstrmsymbols" + fstrmepslocal = kaldi_root / "src/fstbin/fstrmepslocal" + fstminimizeencoded = kaldi_root / "src/fstbin/fstminimizeencoded" + + try: + with open(hlga_graph, "wb") as out_f: + res = subprocess.run( + [ + fsttablecompose, + h_graph, + lg_graph, + ], + capture_output=True, + check=True, + ) + res = subprocess.run( + [fstdeterminizestar, "--use-log=true"], + input=res.stdout, + capture_output=True, + check=True, + ) + res = subprocess.run( + [fstrmsymbols, disambig_in_words_file_int], + input=res.stdout, + capture_output=True, + check=True, + ) + res = subprocess.run( + [fstrmepslocal], + input=res.stdout, + capture_output=True, + check=True, + ) + res = subprocess.run( + [fstminimizeencoded], + input=res.stdout, + capture_output=True, + check=True, + ) + out_f.write(res.stdout) + except subprocess.CalledProcessError as e: + logger.error(f"cmd: {e.cmd}, err: {e.stderr.decode('utf-8')}") + os.remove(hlga_graph) + raise + + return hlga_graph + + +def create_HLa( + kaldi_root: Path, + fst_dir: Path, + unique_label: str, + h_graph: Path, + l_graph: Path, + disambig_in_words_file_int: Path, +) -> Path: + hla_graph = fst_dir / f"HLa.{unique_label}.fst" + + if not hla_graph.exists(): + logger.info(f"Creating {hla_graph}") + + fsttablecompose = kaldi_root / "src/fstbin/fsttablecompose" + fstdeterminizestar = kaldi_root / "src/fstbin/fstdeterminizestar" + fstrmsymbols = kaldi_root / "src/fstbin/fstrmsymbols" + fstrmepslocal = kaldi_root / "src/fstbin/fstrmepslocal" + fstminimizeencoded = kaldi_root / "src/fstbin/fstminimizeencoded" + + try: + with open(hla_graph, "wb") as out_f: + res = subprocess.run( + [ + fsttablecompose, + h_graph, + l_graph, + ], + capture_output=True, + check=True, + ) + res = subprocess.run( + [fstdeterminizestar, "--use-log=true"], + input=res.stdout, + capture_output=True, + check=True, + ) + res = subprocess.run( + [fstrmsymbols, disambig_in_words_file_int], + input=res.stdout, + capture_output=True, + check=True, + ) + res = subprocess.run( + [fstrmepslocal], + input=res.stdout, + capture_output=True, + check=True, + ) + res = subprocess.run( + [fstminimizeencoded], + input=res.stdout, + capture_output=True, + check=True, + ) + out_f.write(res.stdout) + except subprocess.CalledProcessError as e: + logger.error(f"cmd: {e.cmd}, err: {e.stderr.decode('utf-8')}") + os.remove(hla_graph) + raise + + return hla_graph + + +def create_HLG( + kaldi_root: Path, + fst_dir: Path, + unique_label: str, + hlga_graph: Path, + prefix: str = "HLG", +) -> Path: + hlg_graph = fst_dir / f"{prefix}.{unique_label}.fst" + + if not hlg_graph.exists(): + logger.info(f"Creating {hlg_graph}") + + add_self_loop = script_dir / "add-self-loop-simple" + kaldi_src = kaldi_root / "src" + kaldi_lib = kaldi_src / "lib" + + try: + if not add_self_loop.exists(): + fst_include = kaldi_root / "tools/openfst-1.6.7/include" + add_self_loop_src = script_dir / "add-self-loop-simple.cc" + + subprocess.run( + [ + "c++", + f"-I{kaldi_src}", + f"-I{fst_include}", + f"-L{kaldi_lib}", + add_self_loop_src, + "-lkaldi-base", + "-lkaldi-fstext", + "-o", + add_self_loop, + ], + check=True, + ) + + my_env = os.environ.copy() + my_env["LD_LIBRARY_PATH"] = f"{kaldi_lib}:{my_env['LD_LIBRARY_PATH']}" + + subprocess.run( + [ + add_self_loop, + hlga_graph, + hlg_graph, + ], + check=True, + capture_output=True, + env=my_env, + ) + except subprocess.CalledProcessError as e: + logger.error(f"cmd: {e.cmd}, err: {e.stderr.decode('utf-8')}") + raise + + return hlg_graph + + +def initalize_kaldi(cfg: KaldiInitializerConfig) -> Path: + if cfg.fst_dir is None: + cfg.fst_dir = osp.join(cfg.data_dir, "kaldi") + if cfg.out_labels is None: + cfg.out_labels = cfg.in_labels + + kaldi_root = Path(cfg.kaldi_root) + data_dir = Path(cfg.data_dir) + fst_dir = Path(cfg.fst_dir) + fst_dir.mkdir(parents=True, exist_ok=True) + + arpa_base = osp.splitext(osp.basename(cfg.lm_arpa))[0] + unique_label = f"{cfg.in_labels}.{arpa_base}" + + with open(data_dir / f"dict.{cfg.in_labels}.txt", "r") as f: + vocab = Dictionary.load(f) + + in_units_file = create_units(fst_dir, cfg.in_labels, vocab) + + grammar_graph, out_words_file = create_G( + kaldi_root, fst_dir, Path(cfg.lm_arpa), arpa_base + ) + + disambig_lexicon_file, disambig_L_in_units_file = create_lexicon( + cfg, fst_dir, unique_label, in_units_file, out_words_file + ) + + h_graph, h_out_units_file, disambig_in_units_file_int = create_H( + kaldi_root, + fst_dir, + disambig_L_in_units_file, + cfg.in_labels, + vocab, + cfg.blank_symbol, + cfg.silence_symbol, + ) + lexicon_graph = create_L( + kaldi_root, + fst_dir, + unique_label, + disambig_lexicon_file, + disambig_L_in_units_file, + out_words_file, + ) + lg_graph = create_LG( + kaldi_root, fst_dir, unique_label, lexicon_graph, grammar_graph + ) + hlga_graph = create_HLGa( + kaldi_root, fst_dir, unique_label, h_graph, lg_graph, disambig_in_units_file_int + ) + hlg_graph = create_HLG(kaldi_root, fst_dir, unique_label, hlga_graph) + + # for debugging + # hla_graph = create_HLa(kaldi_root, fst_dir, unique_label, h_graph, lexicon_graph, disambig_in_units_file_int) + # hl_graph = create_HLG(kaldi_root, fst_dir, unique_label, hla_graph, prefix="HL_looped") + # create_HLG(kaldi_root, fst_dir, "phnc", h_graph, prefix="H_looped") + + return hlg_graph + + +@hydra.main(config_path=config_path, config_name="kaldi_initializer") +def cli_main(cfg: KaldiInitializerConfig) -> None: + container = OmegaConf.to_container(cfg, resolve=True, enum_to_str=True) + cfg = OmegaConf.create(container) + OmegaConf.set_struct(cfg, True) + initalize_kaldi(cfg) + + +if __name__ == "__main__": + + logging.root.setLevel(logging.INFO) + logging.basicConfig(level=logging.INFO) + + try: + from hydra._internal.utils import ( + get_args, + ) # pylint: disable=import-outside-toplevel + + cfg_name = get_args().config_name or "kaldi_initializer" + except ImportError: + logger.warning("Failed to get config name from hydra args") + cfg_name = "kaldi_initializer" + + cs = ConfigStore.instance() + cs.store(name=cfg_name, node=KaldiInitializerConfig) + + cli_main() diff --git a/data/fairseq/examples/speech_recognition/models/__init__.py b/data/fairseq/examples/speech_recognition/models/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..54b5a1c31243e55d384f80ef9514461cd35b15c6 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/models/__init__.py @@ -0,0 +1,8 @@ +import importlib +import os + + +for file in sorted(os.listdir(os.path.dirname(__file__))): + if file.endswith(".py") and not file.startswith("_"): + model_name = file[: file.find(".py")] + importlib.import_module("examples.speech_recognition.models." + model_name) diff --git a/data/fairseq/examples/speech_recognition/models/vggtransformer.py b/data/fairseq/examples/speech_recognition/models/vggtransformer.py new file mode 100644 index 0000000000000000000000000000000000000000..bca0ae59a8cbe2b7c337e395021c883a61d101ee --- /dev/null +++ b/data/fairseq/examples/speech_recognition/models/vggtransformer.py @@ -0,0 +1,1020 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import argparse +import math +from collections.abc import Iterable + +import torch +import torch.nn as nn +from examples.speech_recognition.data.data_utils import lengths_to_encoder_padding_mask +from fairseq import utils +from fairseq.models import ( + FairseqEncoder, + FairseqEncoderDecoderModel, + FairseqEncoderModel, + FairseqIncrementalDecoder, + register_model, + register_model_architecture, +) +from fairseq.modules import ( + LinearizedConvolution, + TransformerDecoderLayer, + TransformerEncoderLayer, + VGGBlock, +) + + +@register_model("asr_vggtransformer") +class VGGTransformerModel(FairseqEncoderDecoderModel): + """ + Transformers with convolutional context for ASR + https://arxiv.org/abs/1904.11660 + """ + + def __init__(self, encoder, decoder): + super().__init__(encoder, decoder) + + @staticmethod + def add_args(parser): + """Add model-specific arguments to the parser.""" + parser.add_argument( + "--input-feat-per-channel", + type=int, + metavar="N", + help="encoder input dimension per input channel", + ) + parser.add_argument( + "--vggblock-enc-config", + type=str, + metavar="EXPR", + help=""" + an array of tuples each containing the configuration of one vggblock: + [(out_channels, + conv_kernel_size, + pooling_kernel_size, + num_conv_layers, + use_layer_norm), ...]) + """, + ) + parser.add_argument( + "--transformer-enc-config", + type=str, + metavar="EXPR", + help="""" + a tuple containing the configuration of the encoder transformer layers + configurations: + [(input_dim, + num_heads, + ffn_dim, + normalize_before, + dropout, + attention_dropout, + relu_dropout), ...]') + """, + ) + parser.add_argument( + "--enc-output-dim", + type=int, + metavar="N", + help=""" + encoder output dimension, can be None. If specified, projecting the + transformer output to the specified dimension""", + ) + parser.add_argument( + "--in-channels", + type=int, + metavar="N", + help="number of encoder input channels", + ) + parser.add_argument( + "--tgt-embed-dim", + type=int, + metavar="N", + help="embedding dimension of the decoder target tokens", + ) + parser.add_argument( + "--transformer-dec-config", + type=str, + metavar="EXPR", + help=""" + a tuple containing the configuration of the decoder transformer layers + configurations: + [(input_dim, + num_heads, + ffn_dim, + normalize_before, + dropout, + attention_dropout, + relu_dropout), ...] + """, + ) + parser.add_argument( + "--conv-dec-config", + type=str, + metavar="EXPR", + help=""" + an array of tuples for the decoder 1-D convolution config + [(out_channels, conv_kernel_size, use_layer_norm), ...]""", + ) + + @classmethod + def build_encoder(cls, args, task): + return VGGTransformerEncoder( + input_feat_per_channel=args.input_feat_per_channel, + vggblock_config=eval(args.vggblock_enc_config), + transformer_config=eval(args.transformer_enc_config), + encoder_output_dim=args.enc_output_dim, + in_channels=args.in_channels, + ) + + @classmethod + def build_decoder(cls, args, task): + return TransformerDecoder( + dictionary=task.target_dictionary, + embed_dim=args.tgt_embed_dim, + transformer_config=eval(args.transformer_dec_config), + conv_config=eval(args.conv_dec_config), + encoder_output_dim=args.enc_output_dim, + ) + + @classmethod + def build_model(cls, args, task): + """Build a new model instance.""" + # make sure that all args are properly defaulted + # (in case there are any new ones) + base_architecture(args) + + encoder = cls.build_encoder(args, task) + decoder = cls.build_decoder(args, task) + return cls(encoder, decoder) + + def get_normalized_probs(self, net_output, log_probs, sample=None): + # net_output['encoder_out'] is a (B, T, D) tensor + lprobs = super().get_normalized_probs(net_output, log_probs, sample) + lprobs.batch_first = True + return lprobs + + +DEFAULT_ENC_VGGBLOCK_CONFIG = ((32, 3, 2, 2, False),) * 2 +DEFAULT_ENC_TRANSFORMER_CONFIG = ((256, 4, 1024, True, 0.2, 0.2, 0.2),) * 2 +# 256: embedding dimension +# 4: number of heads +# 1024: FFN +# True: apply layerNorm before (dropout + resiaul) instead of after +# 0.2 (dropout): dropout after MultiheadAttention and second FC +# 0.2 (attention_dropout): dropout in MultiheadAttention +# 0.2 (relu_dropout): dropout after ReLu +DEFAULT_DEC_TRANSFORMER_CONFIG = ((256, 2, 1024, True, 0.2, 0.2, 0.2),) * 2 +DEFAULT_DEC_CONV_CONFIG = ((256, 3, True),) * 2 + + +# TODO: repace transformer encoder config from one liner +# to explicit args to get rid of this transformation +def prepare_transformer_encoder_params( + input_dim, + num_heads, + ffn_dim, + normalize_before, + dropout, + attention_dropout, + relu_dropout, +): + args = argparse.Namespace() + args.encoder_embed_dim = input_dim + args.encoder_attention_heads = num_heads + args.attention_dropout = attention_dropout + args.dropout = dropout + args.activation_dropout = relu_dropout + args.encoder_normalize_before = normalize_before + args.encoder_ffn_embed_dim = ffn_dim + return args + + +def prepare_transformer_decoder_params( + input_dim, + num_heads, + ffn_dim, + normalize_before, + dropout, + attention_dropout, + relu_dropout, +): + args = argparse.Namespace() + args.encoder_embed_dim = None + args.decoder_embed_dim = input_dim + args.decoder_attention_heads = num_heads + args.attention_dropout = attention_dropout + args.dropout = dropout + args.activation_dropout = relu_dropout + args.decoder_normalize_before = normalize_before + args.decoder_ffn_embed_dim = ffn_dim + return args + + +class VGGTransformerEncoder(FairseqEncoder): + """VGG + Transformer encoder""" + + def __init__( + self, + input_feat_per_channel, + vggblock_config=DEFAULT_ENC_VGGBLOCK_CONFIG, + transformer_config=DEFAULT_ENC_TRANSFORMER_CONFIG, + encoder_output_dim=512, + in_channels=1, + transformer_context=None, + transformer_sampling=None, + ): + """constructor for VGGTransformerEncoder + + Args: + - input_feat_per_channel: feature dim (not including stacked, + just base feature) + - in_channel: # input channels (e.g., if stack 8 feature vector + together, this is 8) + - vggblock_config: configuration of vggblock, see comments on + DEFAULT_ENC_VGGBLOCK_CONFIG + - transformer_config: configuration of transformer layer, see comments + on DEFAULT_ENC_TRANSFORMER_CONFIG + - encoder_output_dim: final transformer output embedding dimension + - transformer_context: (left, right) if set, self-attention will be focused + on (t-left, t+right) + - transformer_sampling: an iterable of int, must match with + len(transformer_config), transformer_sampling[i] indicates sampling + factor for i-th transformer layer, after multihead att and feedfoward + part + """ + super().__init__(None) + + self.num_vggblocks = 0 + if vggblock_config is not None: + if not isinstance(vggblock_config, Iterable): + raise ValueError("vggblock_config is not iterable") + self.num_vggblocks = len(vggblock_config) + + self.conv_layers = nn.ModuleList() + self.in_channels = in_channels + self.input_dim = input_feat_per_channel + self.pooling_kernel_sizes = [] + + if vggblock_config is not None: + for _, config in enumerate(vggblock_config): + ( + out_channels, + conv_kernel_size, + pooling_kernel_size, + num_conv_layers, + layer_norm, + ) = config + self.conv_layers.append( + VGGBlock( + in_channels, + out_channels, + conv_kernel_size, + pooling_kernel_size, + num_conv_layers, + input_dim=input_feat_per_channel, + layer_norm=layer_norm, + ) + ) + self.pooling_kernel_sizes.append(pooling_kernel_size) + in_channels = out_channels + input_feat_per_channel = self.conv_layers[-1].output_dim + + transformer_input_dim = self.infer_conv_output_dim( + self.in_channels, self.input_dim + ) + # transformer_input_dim is the output dimension of VGG part + + self.validate_transformer_config(transformer_config) + self.transformer_context = self.parse_transformer_context(transformer_context) + self.transformer_sampling = self.parse_transformer_sampling( + transformer_sampling, len(transformer_config) + ) + + self.transformer_layers = nn.ModuleList() + + if transformer_input_dim != transformer_config[0][0]: + self.transformer_layers.append( + Linear(transformer_input_dim, transformer_config[0][0]) + ) + self.transformer_layers.append( + TransformerEncoderLayer( + prepare_transformer_encoder_params(*transformer_config[0]) + ) + ) + + for i in range(1, len(transformer_config)): + if transformer_config[i - 1][0] != transformer_config[i][0]: + self.transformer_layers.append( + Linear(transformer_config[i - 1][0], transformer_config[i][0]) + ) + self.transformer_layers.append( + TransformerEncoderLayer( + prepare_transformer_encoder_params(*transformer_config[i]) + ) + ) + + self.encoder_output_dim = encoder_output_dim + self.transformer_layers.extend( + [ + Linear(transformer_config[-1][0], encoder_output_dim), + LayerNorm(encoder_output_dim), + ] + ) + + def forward(self, src_tokens, src_lengths, **kwargs): + """ + src_tokens: padded tensor (B, T, C * feat) + src_lengths: tensor of original lengths of input utterances (B,) + """ + bsz, max_seq_len, _ = src_tokens.size() + x = src_tokens.view(bsz, max_seq_len, self.in_channels, self.input_dim) + x = x.transpose(1, 2).contiguous() + # (B, C, T, feat) + + for layer_idx in range(len(self.conv_layers)): + x = self.conv_layers[layer_idx](x) + + bsz, _, output_seq_len, _ = x.size() + + # (B, C, T, feat) -> (B, T, C, feat) -> (T, B, C, feat) -> (T, B, C * feat) + x = x.transpose(1, 2).transpose(0, 1) + x = x.contiguous().view(output_seq_len, bsz, -1) + + input_lengths = src_lengths.clone() + for s in self.pooling_kernel_sizes: + input_lengths = (input_lengths.float() / s).ceil().long() + + encoder_padding_mask, _ = lengths_to_encoder_padding_mask( + input_lengths, batch_first=True + ) + if not encoder_padding_mask.any(): + encoder_padding_mask = None + + subsampling_factor = int(max_seq_len * 1.0 / output_seq_len + 0.5) + attn_mask = self.lengths_to_attn_mask(input_lengths, subsampling_factor) + + transformer_layer_idx = 0 + + for layer_idx in range(len(self.transformer_layers)): + + if isinstance(self.transformer_layers[layer_idx], TransformerEncoderLayer): + x = self.transformer_layers[layer_idx]( + x, encoder_padding_mask, attn_mask + ) + + if self.transformer_sampling[transformer_layer_idx] != 1: + sampling_factor = self.transformer_sampling[transformer_layer_idx] + x, encoder_padding_mask, attn_mask = self.slice( + x, encoder_padding_mask, attn_mask, sampling_factor + ) + + transformer_layer_idx += 1 + + else: + x = self.transformer_layers[layer_idx](x) + + # encoder_padding_maks is a (T x B) tensor, its [t, b] elements indicate + # whether encoder_output[t, b] is valid or not (valid=0, invalid=1) + + return { + "encoder_out": x, # (T, B, C) + "encoder_padding_mask": encoder_padding_mask.t() + if encoder_padding_mask is not None + else None, + # (B, T) --> (T, B) + } + + def infer_conv_output_dim(self, in_channels, input_dim): + sample_seq_len = 200 + sample_bsz = 10 + x = torch.randn(sample_bsz, in_channels, sample_seq_len, input_dim) + for i, _ in enumerate(self.conv_layers): + x = self.conv_layers[i](x) + x = x.transpose(1, 2) + mb, seq = x.size()[:2] + return x.contiguous().view(mb, seq, -1).size(-1) + + def validate_transformer_config(self, transformer_config): + for config in transformer_config: + input_dim, num_heads = config[:2] + if input_dim % num_heads != 0: + msg = ( + "ERROR in transformer config {}: ".format(config) + + "input dimension {} ".format(input_dim) + + "not dividable by number of heads {}".format(num_heads) + ) + raise ValueError(msg) + + def parse_transformer_context(self, transformer_context): + """ + transformer_context can be the following: + - None; indicates no context is used, i.e., + transformer can access full context + - a tuple/list of two int; indicates left and right context, + any number <0 indicates infinite context + * e.g., (5, 6) indicates that for query at x_t, transformer can + access [t-5, t+6] (inclusive) + * e.g., (-1, 6) indicates that for query at x_t, transformer can + access [0, t+6] (inclusive) + """ + if transformer_context is None: + return None + + if not isinstance(transformer_context, Iterable): + raise ValueError("transformer context must be Iterable if it is not None") + + if len(transformer_context) != 2: + raise ValueError("transformer context must have length 2") + + left_context = transformer_context[0] + if left_context < 0: + left_context = None + + right_context = transformer_context[1] + if right_context < 0: + right_context = None + + if left_context is None and right_context is None: + return None + + return (left_context, right_context) + + def parse_transformer_sampling(self, transformer_sampling, num_layers): + """ + parsing transformer sampling configuration + + Args: + - transformer_sampling, accepted input: + * None, indicating no sampling + * an Iterable with int (>0) as element + - num_layers, expected number of transformer layers, must match with + the length of transformer_sampling if it is not None + + Returns: + - A tuple with length num_layers + """ + if transformer_sampling is None: + return (1,) * num_layers + + if not isinstance(transformer_sampling, Iterable): + raise ValueError( + "transformer_sampling must be an iterable if it is not None" + ) + + if len(transformer_sampling) != num_layers: + raise ValueError( + "transformer_sampling {} does not match with the number " + "of layers {}".format(transformer_sampling, num_layers) + ) + + for layer, value in enumerate(transformer_sampling): + if not isinstance(value, int): + raise ValueError("Invalid value in transformer_sampling: ") + if value < 1: + raise ValueError( + "{} layer's subsampling is {}.".format(layer, value) + + " This is not allowed! " + ) + return transformer_sampling + + def slice(self, embedding, padding_mask, attn_mask, sampling_factor): + """ + embedding is a (T, B, D) tensor + padding_mask is a (B, T) tensor or None + attn_mask is a (T, T) tensor or None + """ + embedding = embedding[::sampling_factor, :, :] + if padding_mask is not None: + padding_mask = padding_mask[:, ::sampling_factor] + if attn_mask is not None: + attn_mask = attn_mask[::sampling_factor, ::sampling_factor] + + return embedding, padding_mask, attn_mask + + def lengths_to_attn_mask(self, input_lengths, subsampling_factor=1): + """ + create attention mask according to sequence lengths and transformer + context + + Args: + - input_lengths: (B, )-shape Int/Long tensor; input_lengths[b] is + the length of b-th sequence + - subsampling_factor: int + * Note that the left_context and right_context is specified in + the input frame-level while input to transformer may already + go through subsampling (e.g., the use of striding in vggblock) + we use subsampling_factor to scale the left/right context + + Return: + - a (T, T) binary tensor or None, where T is max(input_lengths) + * if self.transformer_context is None, None + * if left_context is None, + * attn_mask[t, t + right_context + 1:] = 1 + * others = 0 + * if right_context is None, + * attn_mask[t, 0:t - left_context] = 1 + * others = 0 + * elsif + * attn_mask[t, t - left_context: t + right_context + 1] = 0 + * others = 1 + """ + if self.transformer_context is None: + return None + + maxT = torch.max(input_lengths).item() + attn_mask = torch.zeros(maxT, maxT) + + left_context = self.transformer_context[0] + right_context = self.transformer_context[1] + if left_context is not None: + left_context = math.ceil(self.transformer_context[0] / subsampling_factor) + if right_context is not None: + right_context = math.ceil(self.transformer_context[1] / subsampling_factor) + + for t in range(maxT): + if left_context is not None: + st = 0 + en = max(st, t - left_context) + attn_mask[t, st:en] = 1 + if right_context is not None: + st = t + right_context + 1 + st = min(st, maxT - 1) + attn_mask[t, st:] = 1 + + return attn_mask.to(input_lengths.device) + + def reorder_encoder_out(self, encoder_out, new_order): + encoder_out["encoder_out"] = encoder_out["encoder_out"].index_select( + 1, new_order + ) + if encoder_out["encoder_padding_mask"] is not None: + encoder_out["encoder_padding_mask"] = encoder_out[ + "encoder_padding_mask" + ].index_select(1, new_order) + return encoder_out + + +class TransformerDecoder(FairseqIncrementalDecoder): + """ + Transformer decoder consisting of *args.decoder_layers* layers. Each layer + is a :class:`TransformerDecoderLayer`. + Args: + args (argparse.Namespace): parsed command-line arguments + dictionary (~fairseq.data.Dictionary): decoding dictionary + embed_tokens (torch.nn.Embedding): output embedding + no_encoder_attn (bool, optional): whether to attend to encoder outputs. + Default: ``False`` + left_pad (bool, optional): whether the input is left-padded. Default: + ``False`` + """ + + def __init__( + self, + dictionary, + embed_dim=512, + transformer_config=DEFAULT_ENC_TRANSFORMER_CONFIG, + conv_config=DEFAULT_DEC_CONV_CONFIG, + encoder_output_dim=512, + ): + + super().__init__(dictionary) + vocab_size = len(dictionary) + self.padding_idx = dictionary.pad() + self.embed_tokens = Embedding(vocab_size, embed_dim, self.padding_idx) + + self.conv_layers = nn.ModuleList() + for i in range(len(conv_config)): + out_channels, kernel_size, layer_norm = conv_config[i] + if i == 0: + conv_layer = LinearizedConv1d( + embed_dim, out_channels, kernel_size, padding=kernel_size - 1 + ) + else: + conv_layer = LinearizedConv1d( + conv_config[i - 1][0], + out_channels, + kernel_size, + padding=kernel_size - 1, + ) + self.conv_layers.append(conv_layer) + if layer_norm: + self.conv_layers.append(nn.LayerNorm(out_channels)) + self.conv_layers.append(nn.ReLU()) + + self.layers = nn.ModuleList() + if conv_config[-1][0] != transformer_config[0][0]: + self.layers.append(Linear(conv_config[-1][0], transformer_config[0][0])) + self.layers.append( + TransformerDecoderLayer( + prepare_transformer_decoder_params(*transformer_config[0]) + ) + ) + + for i in range(1, len(transformer_config)): + if transformer_config[i - 1][0] != transformer_config[i][0]: + self.layers.append( + Linear(transformer_config[i - 1][0], transformer_config[i][0]) + ) + self.layers.append( + TransformerDecoderLayer( + prepare_transformer_decoder_params(*transformer_config[i]) + ) + ) + self.fc_out = Linear(transformer_config[-1][0], vocab_size) + + def forward(self, prev_output_tokens, encoder_out=None, incremental_state=None): + """ + Args: + prev_output_tokens (LongTensor): previous decoder outputs of shape + `(batch, tgt_len)`, for input feeding/teacher forcing + encoder_out (Tensor, optional): output from the encoder, used for + encoder-side attention + incremental_state (dict): dictionary used for storing state during + :ref:`Incremental decoding` + Returns: + tuple: + - the last decoder layer's output of shape `(batch, tgt_len, + vocab)` + - the last decoder layer's attention weights of shape `(batch, + tgt_len, src_len)` + """ + target_padding_mask = ( + (prev_output_tokens == self.padding_idx).to(prev_output_tokens.device) + if incremental_state is None + else None + ) + + if incremental_state is not None: + prev_output_tokens = prev_output_tokens[:, -1:] + + # embed tokens + x = self.embed_tokens(prev_output_tokens) + + # B x T x C -> T x B x C + x = self._transpose_if_training(x, incremental_state) + + for layer in self.conv_layers: + if isinstance(layer, LinearizedConvolution): + x = layer(x, incremental_state) + else: + x = layer(x) + + # B x T x C -> T x B x C + x = self._transpose_if_inference(x, incremental_state) + + # decoder layers + for layer in self.layers: + if isinstance(layer, TransformerDecoderLayer): + x, *_ = layer( + x, + (encoder_out["encoder_out"] if encoder_out is not None else None), + ( + encoder_out["encoder_padding_mask"].t() + if encoder_out["encoder_padding_mask"] is not None + else None + ), + incremental_state, + self_attn_mask=( + self.buffered_future_mask(x) + if incremental_state is None + else None + ), + self_attn_padding_mask=( + target_padding_mask if incremental_state is None else None + ), + ) + else: + x = layer(x) + + # T x B x C -> B x T x C + x = x.transpose(0, 1) + + x = self.fc_out(x) + + return x, None + + def buffered_future_mask(self, tensor): + dim = tensor.size(0) + if ( + not hasattr(self, "_future_mask") + or self._future_mask is None + or self._future_mask.device != tensor.device + ): + self._future_mask = torch.triu( + utils.fill_with_neg_inf(tensor.new(dim, dim)), 1 + ) + if self._future_mask.size(0) < dim: + self._future_mask = torch.triu( + utils.fill_with_neg_inf(self._future_mask.resize_(dim, dim)), 1 + ) + return self._future_mask[:dim, :dim] + + def _transpose_if_training(self, x, incremental_state): + if incremental_state is None: + x = x.transpose(0, 1) + return x + + def _transpose_if_inference(self, x, incremental_state): + if incremental_state: + x = x.transpose(0, 1) + return x + + +@register_model("asr_vggtransformer_encoder") +class VGGTransformerEncoderModel(FairseqEncoderModel): + def __init__(self, encoder): + super().__init__(encoder) + + @staticmethod + def add_args(parser): + """Add model-specific arguments to the parser.""" + parser.add_argument( + "--input-feat-per-channel", + type=int, + metavar="N", + help="encoder input dimension per input channel", + ) + parser.add_argument( + "--vggblock-enc-config", + type=str, + metavar="EXPR", + help=""" + an array of tuples each containing the configuration of one vggblock + [(out_channels, conv_kernel_size, pooling_kernel_size,num_conv_layers), ...] + """, + ) + parser.add_argument( + "--transformer-enc-config", + type=str, + metavar="EXPR", + help=""" + a tuple containing the configuration of the Transformer layers + configurations: + [(input_dim, + num_heads, + ffn_dim, + normalize_before, + dropout, + attention_dropout, + relu_dropout), ]""", + ) + parser.add_argument( + "--enc-output-dim", + type=int, + metavar="N", + help="encoder output dimension, projecting the LSTM output", + ) + parser.add_argument( + "--in-channels", + type=int, + metavar="N", + help="number of encoder input channels", + ) + parser.add_argument( + "--transformer-context", + type=str, + metavar="EXPR", + help=""" + either None or a tuple of two ints, indicating left/right context a + transformer can have access to""", + ) + parser.add_argument( + "--transformer-sampling", + type=str, + metavar="EXPR", + help=""" + either None or a tuple of ints, indicating sampling factor in each layer""", + ) + + @classmethod + def build_model(cls, args, task): + """Build a new model instance.""" + base_architecture_enconly(args) + encoder = VGGTransformerEncoderOnly( + vocab_size=len(task.target_dictionary), + input_feat_per_channel=args.input_feat_per_channel, + vggblock_config=eval(args.vggblock_enc_config), + transformer_config=eval(args.transformer_enc_config), + encoder_output_dim=args.enc_output_dim, + in_channels=args.in_channels, + transformer_context=eval(args.transformer_context), + transformer_sampling=eval(args.transformer_sampling), + ) + return cls(encoder) + + def get_normalized_probs(self, net_output, log_probs, sample=None): + # net_output['encoder_out'] is a (T, B, D) tensor + lprobs = super().get_normalized_probs(net_output, log_probs, sample) + # lprobs is a (T, B, D) tensor + # we need to transoose to get (B, T, D) tensor + lprobs = lprobs.transpose(0, 1).contiguous() + lprobs.batch_first = True + return lprobs + + +class VGGTransformerEncoderOnly(VGGTransformerEncoder): + def __init__( + self, + vocab_size, + input_feat_per_channel, + vggblock_config=DEFAULT_ENC_VGGBLOCK_CONFIG, + transformer_config=DEFAULT_ENC_TRANSFORMER_CONFIG, + encoder_output_dim=512, + in_channels=1, + transformer_context=None, + transformer_sampling=None, + ): + super().__init__( + input_feat_per_channel=input_feat_per_channel, + vggblock_config=vggblock_config, + transformer_config=transformer_config, + encoder_output_dim=encoder_output_dim, + in_channels=in_channels, + transformer_context=transformer_context, + transformer_sampling=transformer_sampling, + ) + self.fc_out = Linear(self.encoder_output_dim, vocab_size) + + def forward(self, src_tokens, src_lengths, **kwargs): + """ + src_tokens: padded tensor (B, T, C * feat) + src_lengths: tensor of original lengths of input utterances (B,) + """ + + enc_out = super().forward(src_tokens, src_lengths) + x = self.fc_out(enc_out["encoder_out"]) + # x = F.log_softmax(x, dim=-1) + # Note: no need this line, because model.get_normalized_prob will call + # log_softmax + return { + "encoder_out": x, # (T, B, C) + "encoder_padding_mask": enc_out["encoder_padding_mask"], # (T, B) + } + + def max_positions(self): + """Maximum input length supported by the encoder.""" + return (1e6, 1e6) # an arbitrary large number + + +def Embedding(num_embeddings, embedding_dim, padding_idx): + m = nn.Embedding(num_embeddings, embedding_dim, padding_idx=padding_idx) + # nn.init.uniform_(m.weight, -0.1, 0.1) + # nn.init.constant_(m.weight[padding_idx], 0) + return m + + +def Linear(in_features, out_features, bias=True, dropout=0): + """Linear layer (input: N x T x C)""" + m = nn.Linear(in_features, out_features, bias=bias) + # m.weight.data.uniform_(-0.1, 0.1) + # if bias: + # m.bias.data.uniform_(-0.1, 0.1) + return m + + +def LinearizedConv1d(in_channels, out_channels, kernel_size, dropout=0, **kwargs): + """Weight-normalized Conv1d layer optimized for decoding""" + m = LinearizedConvolution(in_channels, out_channels, kernel_size, **kwargs) + std = math.sqrt((4 * (1.0 - dropout)) / (m.kernel_size[0] * in_channels)) + nn.init.normal_(m.weight, mean=0, std=std) + nn.init.constant_(m.bias, 0) + return nn.utils.weight_norm(m, dim=2) + + +def LayerNorm(embedding_dim): + m = nn.LayerNorm(embedding_dim) + return m + + +# seq2seq models +def base_architecture(args): + args.input_feat_per_channel = getattr(args, "input_feat_per_channel", 40) + args.vggblock_enc_config = getattr( + args, "vggblock_enc_config", DEFAULT_ENC_VGGBLOCK_CONFIG + ) + args.transformer_enc_config = getattr( + args, "transformer_enc_config", DEFAULT_ENC_TRANSFORMER_CONFIG + ) + args.enc_output_dim = getattr(args, "enc_output_dim", 512) + args.in_channels = getattr(args, "in_channels", 1) + args.tgt_embed_dim = getattr(args, "tgt_embed_dim", 128) + args.transformer_dec_config = getattr( + args, "transformer_dec_config", DEFAULT_ENC_TRANSFORMER_CONFIG + ) + args.conv_dec_config = getattr(args, "conv_dec_config", DEFAULT_DEC_CONV_CONFIG) + args.transformer_context = getattr(args, "transformer_context", "None") + + +@register_model_architecture("asr_vggtransformer", "vggtransformer_1") +def vggtransformer_1(args): + args.input_feat_per_channel = getattr(args, "input_feat_per_channel", 80) + args.vggblock_enc_config = getattr( + args, "vggblock_enc_config", "[(64, 3, 2, 2, True), (128, 3, 2, 2, True)]" + ) + args.transformer_enc_config = getattr( + args, + "transformer_enc_config", + "((1024, 16, 4096, True, 0.15, 0.15, 0.15),) * 14", + ) + args.enc_output_dim = getattr(args, "enc_output_dim", 1024) + args.tgt_embed_dim = getattr(args, "tgt_embed_dim", 128) + args.conv_dec_config = getattr(args, "conv_dec_config", "((256, 3, True),) * 4") + args.transformer_dec_config = getattr( + args, + "transformer_dec_config", + "((1024, 16, 4096, True, 0.15, 0.15, 0.15),) * 4", + ) + + +@register_model_architecture("asr_vggtransformer", "vggtransformer_2") +def vggtransformer_2(args): + args.input_feat_per_channel = getattr(args, "input_feat_per_channel", 80) + args.vggblock_enc_config = getattr( + args, "vggblock_enc_config", "[(64, 3, 2, 2, True), (128, 3, 2, 2, True)]" + ) + args.transformer_enc_config = getattr( + args, + "transformer_enc_config", + "((1024, 16, 4096, True, 0.15, 0.15, 0.15),) * 16", + ) + args.enc_output_dim = getattr(args, "enc_output_dim", 1024) + args.tgt_embed_dim = getattr(args, "tgt_embed_dim", 512) + args.conv_dec_config = getattr(args, "conv_dec_config", "((256, 3, True),) * 4") + args.transformer_dec_config = getattr( + args, + "transformer_dec_config", + "((1024, 16, 4096, True, 0.15, 0.15, 0.15),) * 6", + ) + + +@register_model_architecture("asr_vggtransformer", "vggtransformer_base") +def vggtransformer_base(args): + args.input_feat_per_channel = getattr(args, "input_feat_per_channel", 80) + args.vggblock_enc_config = getattr( + args, "vggblock_enc_config", "[(64, 3, 2, 2, True), (128, 3, 2, 2, True)]" + ) + args.transformer_enc_config = getattr( + args, "transformer_enc_config", "((512, 8, 2048, True, 0.15, 0.15, 0.15),) * 12" + ) + + args.enc_output_dim = getattr(args, "enc_output_dim", 512) + args.tgt_embed_dim = getattr(args, "tgt_embed_dim", 512) + args.conv_dec_config = getattr(args, "conv_dec_config", "((256, 3, True),) * 4") + args.transformer_dec_config = getattr( + args, "transformer_dec_config", "((512, 8, 2048, True, 0.15, 0.15, 0.15),) * 6" + ) + # Size estimations: + # Encoder: + # - vggblock param: 64*1*3*3 + 64*64*3*3 + 128*64*3*3 + 128*128*3 = 258K + # Transformer: + # - input dimension adapter: 2560 x 512 -> 1.31M + # - transformer_layers (x12) --> 37.74M + # * MultiheadAttention: 512*512*3 (in_proj) + 512*512 (out_proj) = 1.048M + # * FFN weight: 512*2048*2 = 2.097M + # - output dimension adapter: 512 x 512 -> 0.26 M + # Decoder: + # - LinearizedConv1d: 512 * 256 * 3 + 256 * 256 * 3 * 3 + # - transformer_layer: (x6) --> 25.16M + # * MultiheadAttention (self-attention): 512*512*3 + 512*512 = 1.048M + # * MultiheadAttention (encoder-attention): 512*512*3 + 512*512 = 1.048M + # * FFN: 512*2048*2 = 2.097M + # Final FC: + # - FC: 512*5000 = 256K (assuming vocab size 5K) + # In total: + # ~65 M + + +# CTC models +def base_architecture_enconly(args): + args.input_feat_per_channel = getattr(args, "input_feat_per_channel", 40) + args.vggblock_enc_config = getattr( + args, "vggblock_enc_config", "[(32, 3, 2, 2, True)] * 2" + ) + args.transformer_enc_config = getattr( + args, "transformer_enc_config", "((256, 4, 1024, True, 0.2, 0.2, 0.2),) * 2" + ) + args.enc_output_dim = getattr(args, "enc_output_dim", 512) + args.in_channels = getattr(args, "in_channels", 1) + args.transformer_context = getattr(args, "transformer_context", "None") + args.transformer_sampling = getattr(args, "transformer_sampling", "None") + + +@register_model_architecture("asr_vggtransformer_encoder", "vggtransformer_enc_1") +def vggtransformer_enc_1(args): + # vggtransformer_1 is the same as vggtransformer_enc_big, except the number + # of layers is increased to 16 + # keep it here for backward compatiablity purpose + args.input_feat_per_channel = getattr(args, "input_feat_per_channel", 80) + args.vggblock_enc_config = getattr( + args, "vggblock_enc_config", "[(64, 3, 2, 2, True), (128, 3, 2, 2, True)]" + ) + args.transformer_enc_config = getattr( + args, + "transformer_enc_config", + "((1024, 16, 4096, True, 0.15, 0.15, 0.15),) * 16", + ) + args.enc_output_dim = getattr(args, "enc_output_dim", 1024) diff --git a/data/fairseq/examples/speech_recognition/models/w2l_conv_glu_enc.py b/data/fairseq/examples/speech_recognition/models/w2l_conv_glu_enc.py new file mode 100644 index 0000000000000000000000000000000000000000..655a9b0d19d11e35511392a016f9d6b7d7aa2925 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/models/w2l_conv_glu_enc.py @@ -0,0 +1,177 @@ +#!/usr/bin/env python3 + +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import math + +import torch +import torch.nn as nn +import torch.nn.functional as F +from fairseq.models import ( + FairseqEncoder, + FairseqEncoderModel, + register_model, + register_model_architecture, +) +from fairseq.modules.fairseq_dropout import FairseqDropout + + +default_conv_enc_config = """[ + (400, 13, 170, 0.2), + (440, 14, 0, 0.214), + (484, 15, 0, 0.22898), + (532, 16, 0, 0.2450086), + (584, 17, 0, 0.262159202), + (642, 18, 0, 0.28051034614), + (706, 19, 0, 0.30014607037), + (776, 20, 0, 0.321156295296), + (852, 21, 0, 0.343637235966), + (936, 22, 0, 0.367691842484), + (1028, 23, 0, 0.393430271458), + (1130, 24, 0, 0.42097039046), + (1242, 25, 0, 0.450438317792), + (1366, 26, 0, 0.481969000038), + (1502, 27, 0, 0.51570683004), + (1652, 28, 0, 0.551806308143), + (1816, 29, 0, 0.590432749713), +]""" + + +@register_model("asr_w2l_conv_glu_encoder") +class W2lConvGluEncoderModel(FairseqEncoderModel): + def __init__(self, encoder): + super().__init__(encoder) + + @staticmethod + def add_args(parser): + """Add model-specific arguments to the parser.""" + parser.add_argument( + "--input-feat-per-channel", + type=int, + metavar="N", + help="encoder input dimension per input channel", + ) + parser.add_argument( + "--in-channels", + type=int, + metavar="N", + help="number of encoder input channels", + ) + parser.add_argument( + "--conv-enc-config", + type=str, + metavar="EXPR", + help=""" + an array of tuples each containing the configuration of one conv layer + [(out_channels, kernel_size, padding, dropout), ...] + """, + ) + + @classmethod + def build_model(cls, args, task): + """Build a new model instance.""" + conv_enc_config = getattr(args, "conv_enc_config", default_conv_enc_config) + encoder = W2lConvGluEncoder( + vocab_size=len(task.target_dictionary), + input_feat_per_channel=args.input_feat_per_channel, + in_channels=args.in_channels, + conv_enc_config=eval(conv_enc_config), + ) + return cls(encoder) + + def get_normalized_probs(self, net_output, log_probs, sample=None): + lprobs = super().get_normalized_probs(net_output, log_probs, sample) + lprobs.batch_first = False + return lprobs + + +class W2lConvGluEncoder(FairseqEncoder): + def __init__( + self, vocab_size, input_feat_per_channel, in_channels, conv_enc_config + ): + super().__init__(None) + + self.input_dim = input_feat_per_channel + if in_channels != 1: + raise ValueError("only 1 input channel is currently supported") + + self.conv_layers = nn.ModuleList() + self.linear_layers = nn.ModuleList() + self.dropouts = [] + cur_channels = input_feat_per_channel + + for out_channels, kernel_size, padding, dropout in conv_enc_config: + layer = nn.Conv1d(cur_channels, out_channels, kernel_size, padding=padding) + layer.weight.data.mul_(math.sqrt(3)) # match wav2letter init + self.conv_layers.append(nn.utils.weight_norm(layer)) + self.dropouts.append( + FairseqDropout(dropout, module_name=self.__class__.__name__) + ) + if out_channels % 2 != 0: + raise ValueError("odd # of out_channels is incompatible with GLU") + cur_channels = out_channels // 2 # halved by GLU + + for out_channels in [2 * cur_channels, vocab_size]: + layer = nn.Linear(cur_channels, out_channels) + layer.weight.data.mul_(math.sqrt(3)) + self.linear_layers.append(nn.utils.weight_norm(layer)) + cur_channels = out_channels // 2 + + def forward(self, src_tokens, src_lengths, **kwargs): + + """ + src_tokens: padded tensor (B, T, C * feat) + src_lengths: tensor of original lengths of input utterances (B,) + """ + B, T, _ = src_tokens.size() + x = src_tokens.transpose(1, 2).contiguous() # (B, feat, T) assuming C == 1 + + for layer_idx in range(len(self.conv_layers)): + x = self.conv_layers[layer_idx](x) + x = F.glu(x, dim=1) + x = self.dropouts[layer_idx](x) + + x = x.transpose(1, 2).contiguous() # (B, T, 908) + x = self.linear_layers[0](x) + x = F.glu(x, dim=2) + x = self.dropouts[-1](x) + x = self.linear_layers[1](x) + + assert x.size(0) == B + assert x.size(1) == T + + encoder_out = x.transpose(0, 1) # (T, B, vocab_size) + + # need to debug this -- find a simpler/elegant way in pytorch APIs + encoder_padding_mask = ( + torch.arange(T).view(1, T).expand(B, -1).to(x.device) + >= src_lengths.view(B, 1).expand(-1, T) + ).t() # (B x T) -> (T x B) + + return { + "encoder_out": encoder_out, # (T, B, vocab_size) + "encoder_padding_mask": encoder_padding_mask, # (T, B) + } + + def reorder_encoder_out(self, encoder_out, new_order): + encoder_out["encoder_out"] = encoder_out["encoder_out"].index_select( + 1, new_order + ) + encoder_out["encoder_padding_mask"] = encoder_out[ + "encoder_padding_mask" + ].index_select(1, new_order) + return encoder_out + + def max_positions(self): + """Maximum input length supported by the encoder.""" + return (1e6, 1e6) # an arbitrary large number + + +@register_model_architecture("asr_w2l_conv_glu_encoder", "w2l_conv_glu_enc") +def w2l_conv_glu_enc(args): + args.input_feat_per_channel = getattr(args, "input_feat_per_channel", 80) + args.in_channels = getattr(args, "in_channels", 1) + args.conv_enc_config = getattr(args, "conv_enc_config", default_conv_enc_config) diff --git a/data/fairseq/examples/speech_recognition/new/README.md b/data/fairseq/examples/speech_recognition/new/README.md new file mode 100644 index 0000000000000000000000000000000000000000..5fa0e97245d3ba6db69d11222261b0644960183d --- /dev/null +++ b/data/fairseq/examples/speech_recognition/new/README.md @@ -0,0 +1,43 @@ +# Flashlight Decoder + +This script runs decoding for pre-trained speech recognition models. + +## Usage + +Assuming a few variables: + +```bash +checkpoint= +data= +lm_model= +lexicon= +``` + +Example usage for decoding a fine-tuned Wav2Vec model: + +```bash +python $FAIRSEQ_ROOT/examples/speech_recognition/new/infer.py --multirun \ + task=audio_pretraining \ + task.data=$data \ + task.labels=ltr \ + common_eval.path=$checkpoint \ + decoding.type=kenlm \ + decoding.lexicon=$lexicon \ + decoding.lmpath=$lm_model \ + dataset.gen_subset=dev_clean,dev_other,test_clean,test_other +``` + +Example usage for using Ax to sweep WER parameters (requires `pip install hydra-ax-sweeper`): + +```bash +python $FAIRSEQ_ROOT/examples/speech_recognition/new/infer.py --multirun \ + hydra/sweeper=ax \ + task=audio_pretraining \ + task.data=$data \ + task.labels=ltr \ + common_eval.path=$checkpoint \ + decoding.type=kenlm \ + decoding.lexicon=$lexicon \ + decoding.lmpath=$lm_model \ + dataset.gen_subset=dev_other +``` diff --git a/data/fairseq/examples/speech_recognition/new/__init__.py b/data/fairseq/examples/speech_recognition/new/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/data/fairseq/examples/speech_recognition/new/conf/hydra/sweeper/ax.yaml b/data/fairseq/examples/speech_recognition/new/conf/hydra/sweeper/ax.yaml new file mode 100644 index 0000000000000000000000000000000000000000..38e9c221db234877056f59e5d2881199a3d601b4 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/new/conf/hydra/sweeper/ax.yaml @@ -0,0 +1,29 @@ +# @package hydra.sweeper +_target_: hydra_plugins.hydra_ax_sweeper.ax_sweeper.AxSweeper +max_batch_size: null +ax_config: + max_trials: 128 + early_stop: + minimize: true + max_epochs_without_improvement: 10 + epsilon: 0.025 + experiment: + name: ${dataset.gen_subset} + objective_name: wer + minimize: true + parameter_constraints: null + outcome_constraints: null + status_quo: null + client: + verbose_logging: false + random_seed: null + params: + decoding.lmweight: + type: range + bounds: [0.0, 5.0] + decoding.wordscore: + type: range + bounds: [-5.0, 5.0] + decoding.silweight: + type: range + bounds: [ -8.0, 0.0 ] diff --git a/data/fairseq/examples/speech_recognition/new/conf/hydra/sweeper/ax_sil.yaml b/data/fairseq/examples/speech_recognition/new/conf/hydra/sweeper/ax_sil.yaml new file mode 100644 index 0000000000000000000000000000000000000000..eaaebcf5f606947d6dc520f611e749128c4a009c --- /dev/null +++ b/data/fairseq/examples/speech_recognition/new/conf/hydra/sweeper/ax_sil.yaml @@ -0,0 +1,29 @@ +# @package hydra.sweeper +_target_: hydra_plugins.hydra_ax_sweeper.ax_sweeper.AxSweeper +max_batch_size: null +ax_config: + max_trials: 64 + early_stop: + minimize: true + max_epochs_without_improvement: 10 + epsilon: 0.025 + experiment: + name: ${dataset.gen_subset} + objective_name: wer + minimize: true + parameter_constraints: null + outcome_constraints: null + status_quo: null + client: + verbose_logging: false + random_seed: null + params: + decoding.lmweight: + type: range + bounds: [0.0, 10.0] + decoding.wordscore: + type: range + bounds: [-10.0, 10.0] + decoding.silweight: + type: range + bounds: [ -10.0, 0.0 ] diff --git a/data/fairseq/examples/speech_recognition/new/conf/infer.yaml b/data/fairseq/examples/speech_recognition/new/conf/infer.yaml new file mode 100644 index 0000000000000000000000000000000000000000..2d168d06afbd95c1322c70d065f453a8e967477b --- /dev/null +++ b/data/fairseq/examples/speech_recognition/new/conf/infer.yaml @@ -0,0 +1,27 @@ +# @package _group_ + +defaults: + - task: null + - model: null + +hydra: + run: + dir: ${common_eval.results_path}/${dataset.gen_subset} + sweep: + dir: /checkpoint/${env:USER}/${env:PREFIX}/${common_eval.results_path} + subdir: ${dataset.gen_subset} +common: + user_dir: /private/home/abaevski/fairseq-py/examples/data2vec +common_eval: + results_path: null + path: null + post_process: letter + quiet: true +dataset: + max_tokens: 3000000 + gen_subset: test +distributed_training: + distributed_world_size: 1 +decoding: + beam: 5 + type: viterbi diff --git a/data/fairseq/examples/speech_recognition/new/conf/run_config/fb_slurm_1.yaml b/data/fairseq/examples/speech_recognition/new/conf/run_config/fb_slurm_1.yaml new file mode 100644 index 0000000000000000000000000000000000000000..d0a9b0e5863e0ceae3f5b75049c5357fe101cbcc --- /dev/null +++ b/data/fairseq/examples/speech_recognition/new/conf/run_config/fb_slurm_1.yaml @@ -0,0 +1,28 @@ +# @package _global_ + +hydra: + job: + config: + override_dirname: + kv_sep: ':' + item_sep: '/' + exclude_keys: + - run_config + - distributed_training.distributed_port + - common_eval.path + sweep: + dir: /checkpoint/abaevski/asr/d2v2/decoding/${env:PREFIX}/${hydra.job.config_name}_${hydra.launcher.gpus_per_node}/${hydra.job.override_dirname} +# subdir: ${hydra.job.override_dirname} + launcher: + cpus_per_task: 16 + gpus_per_node: 1 + tasks_per_node: 1 + nodes: 1 + partition: devlab,learnlab + mem_gb: 100 + timeout_min: 2000 + max_num_timeout: 10 + name: ${env:PREFIX}_${hydra.job.config_name} + submitit_folder: ${hydra.sweep.dir}/%j + constraint: volta32gb + exclude: learnfair7598 \ No newline at end of file diff --git a/data/fairseq/examples/speech_recognition/new/conf/run_config/fb_slurm_2g.yaml b/data/fairseq/examples/speech_recognition/new/conf/run_config/fb_slurm_2g.yaml new file mode 100644 index 0000000000000000000000000000000000000000..c0c442f76d06516de3f85675029a8d79e5c4ee69 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/new/conf/run_config/fb_slurm_2g.yaml @@ -0,0 +1,27 @@ +# @package _global_ + +hydra: + job: + config: + override_dirname: + kv_sep: ':' + item_sep: '/' + exclude_keys: + - run_config + - distributed_training.distributed_port + - common_eval.path + sweep: + dir: /checkpoint/abaevski/asr/d2v2/decoding/${env:PREFIX}/${hydra.job.config_name}_${hydra.launcher.gpus_per_node}/${hydra.job.override_dirname} +# subdir: ${hydra.job.override_dirname} + launcher: + cpus_per_task: 16 + gpus_per_node: 2 + tasks_per_node: 2 + nodes: 1 + partition: devlab,learnlab + mem_gb: 100 + timeout_min: 2000 + max_num_timeout: 10 + name: ${env:PREFIX}_${hydra.job.config_name} + submitit_folder: ${hydra.sweep.dir}/%j + constraint: volta32gb \ No newline at end of file diff --git a/data/fairseq/examples/speech_recognition/new/decoders/__init__.py b/data/fairseq/examples/speech_recognition/new/decoders/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/data/fairseq/examples/speech_recognition/new/decoders/base_decoder.py b/data/fairseq/examples/speech_recognition/new/decoders/base_decoder.py new file mode 100644 index 0000000000000000000000000000000000000000..a097969b3c0650cf8ea2ab5f8e96bbc68ea9b97f --- /dev/null +++ b/data/fairseq/examples/speech_recognition/new/decoders/base_decoder.py @@ -0,0 +1,62 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import itertools as it +from typing import Any, Dict, List + +import torch +from fairseq.data.dictionary import Dictionary +from fairseq.models.fairseq_model import FairseqModel + + +class BaseDecoder: + def __init__(self, tgt_dict: Dictionary) -> None: + self.tgt_dict = tgt_dict + self.vocab_size = len(tgt_dict) + + self.blank = ( + tgt_dict.index("") + if "" in tgt_dict.indices + else tgt_dict.bos() + ) + if "" in tgt_dict.indices: + self.silence = tgt_dict.index("") + elif "|" in tgt_dict.indices: + self.silence = tgt_dict.index("|") + else: + self.silence = tgt_dict.eos() + + def generate( + self, models: List[FairseqModel], sample: Dict[str, Any], **unused + ) -> List[List[Dict[str, torch.LongTensor]]]: + encoder_input = { + k: v for k, v in sample["net_input"].items() if k != "prev_output_tokens" + } + emissions = self.get_emissions(models, encoder_input) + return self.decode(emissions) + + def get_emissions( + self, + models: List[FairseqModel], + encoder_input: Dict[str, Any], + ) -> torch.FloatTensor: + model = models[0] + encoder_out = model(**encoder_input) + if hasattr(model, "get_logits"): + emissions = model.get_logits(encoder_out) + else: + emissions = model.get_normalized_probs(encoder_out, log_probs=True) + return emissions.transpose(0, 1).float().cpu().contiguous() + + def get_tokens(self, idxs: torch.IntTensor) -> torch.LongTensor: + idxs = (g[0] for g in it.groupby(idxs)) + idxs = filter(lambda x: x != self.blank, idxs) + return torch.LongTensor(list(idxs)) + + def decode( + self, + emissions: torch.FloatTensor, + ) -> List[List[Dict[str, torch.LongTensor]]]: + raise NotImplementedError diff --git a/data/fairseq/examples/speech_recognition/new/decoders/decoder.py b/data/fairseq/examples/speech_recognition/new/decoders/decoder.py new file mode 100644 index 0000000000000000000000000000000000000000..b5bec8cf707b53104ef7a45993a5db2893d3443b --- /dev/null +++ b/data/fairseq/examples/speech_recognition/new/decoders/decoder.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python3 + +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +from typing import Union + +from fairseq.data.dictionary import Dictionary + +from .decoder_config import DecoderConfig, FlashlightDecoderConfig +from .base_decoder import BaseDecoder + + +def Decoder( + cfg: Union[DecoderConfig, FlashlightDecoderConfig], tgt_dict: Dictionary +) -> BaseDecoder: + + if cfg.type == "viterbi": + from .viterbi_decoder import ViterbiDecoder + + return ViterbiDecoder(tgt_dict) + if cfg.type == "kenlm": + from .flashlight_decoder import KenLMDecoder + + return KenLMDecoder(cfg, tgt_dict) + if cfg.type == "fairseqlm": + from .flashlight_decoder import FairseqLMDecoder + + return FairseqLMDecoder(cfg, tgt_dict) + raise NotImplementedError(f"Invalid decoder name: {cfg.name}") diff --git a/data/fairseq/examples/speech_recognition/new/decoders/decoder_config.py b/data/fairseq/examples/speech_recognition/new/decoders/decoder_config.py new file mode 100644 index 0000000000000000000000000000000000000000..659eb94a9b8187a7c126d7b439ac2742f9d72022 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/new/decoders/decoder_config.py @@ -0,0 +1,70 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import math +from dataclasses import dataclass, field +from typing import Optional + +from fairseq.dataclass.configs import FairseqDataclass +from fairseq.dataclass.constants import ChoiceEnum +from omegaconf import MISSING + + +DECODER_CHOICES = ChoiceEnum(["viterbi", "kenlm", "fairseqlm"]) + + +@dataclass +class DecoderConfig(FairseqDataclass): + type: DECODER_CHOICES = field( + default="viterbi", + metadata={"help": "The type of decoder to use"}, + ) + + +@dataclass +class FlashlightDecoderConfig(FairseqDataclass): + nbest: int = field( + default=1, + metadata={"help": "Number of decodings to return"}, + ) + unitlm: bool = field( + default=False, + metadata={"help": "If set, use unit language model"}, + ) + lmpath: str = field( + default=MISSING, + metadata={"help": "Language model for KenLM decoder"}, + ) + lexicon: Optional[str] = field( + default=None, + metadata={"help": "Lexicon for Flashlight decoder"}, + ) + beam: int = field( + default=50, + metadata={"help": "Number of beams to use for decoding"}, + ) + beamthreshold: float = field( + default=50.0, + metadata={"help": "Threshold for beam search decoding"}, + ) + beamsizetoken: Optional[int] = field( + default=None, metadata={"help": "Beam size to use"} + ) + wordscore: float = field( + default=-1, + metadata={"help": "Word score for KenLM decoder"}, + ) + unkweight: float = field( + default=-math.inf, + metadata={"help": "Unknown weight for KenLM decoder"}, + ) + silweight: float = field( + default=0, + metadata={"help": "Silence weight for KenLM decoder"}, + ) + lmweight: float = field( + default=2, + metadata={"help": "Weight for LM while interpolating score"}, + ) diff --git a/data/fairseq/examples/speech_recognition/new/decoders/flashlight_decoder.py b/data/fairseq/examples/speech_recognition/new/decoders/flashlight_decoder.py new file mode 100644 index 0000000000000000000000000000000000000000..7790fcdb8202101d27bf6798554ed7bc25f5c1a6 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/new/decoders/flashlight_decoder.py @@ -0,0 +1,433 @@ +#!/usr/bin/env python3 + +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import gc +import os.path as osp +import warnings +from collections import deque, namedtuple +from typing import Any, Dict, Tuple + +import numpy as np +import torch +from fairseq import tasks +from fairseq.data.dictionary import Dictionary +from fairseq.dataclass.utils import convert_namespace_to_omegaconf +from fairseq.models.fairseq_model import FairseqModel +from fairseq.utils import apply_to_sample +from omegaconf import open_dict, OmegaConf + +from typing import List + +from .decoder_config import FlashlightDecoderConfig +from .base_decoder import BaseDecoder + +try: + from flashlight.lib.text.decoder import ( + LM, + CriterionType, + DecodeResult, + KenLM, + LexiconDecoder, + LexiconDecoderOptions, + LexiconFreeDecoder, + LexiconFreeDecoderOptions, + LMState, + SmearingMode, + Trie, + ) + from flashlight.lib.text.dictionary import create_word_dict, load_words + from flashlight.lib.text.dictionary import Dictionary as flDictionary +except ImportError: + warnings.warn( + "flashlight python bindings are required to use this functionality. " + "Please install from " + "https://github.com/facebookresearch/flashlight/tree/master/bindings/python" + ) + LM = object + LMState = object + + +class KenLMDecoder(BaseDecoder): + def __init__(self, cfg: FlashlightDecoderConfig, tgt_dict: Dictionary) -> None: + super().__init__(tgt_dict) + + self.nbest = cfg.nbest + self.unitlm = cfg.unitlm + + if cfg.lexicon: + self.lexicon = load_words(cfg.lexicon) + self.word_dict = create_word_dict(self.lexicon) + self.unk_word = self.word_dict.get_index("") + + self.lm = KenLM(cfg.lmpath, self.word_dict) + self.trie = Trie(self.vocab_size, self.silence) + + start_state = self.lm.start(False) + for word, spellings in self.lexicon.items(): + word_idx = self.word_dict.get_index(word) + _, score = self.lm.score(start_state, word_idx) + for spelling in spellings: + spelling_idxs = [tgt_dict.index(token) for token in spelling] + assert ( + tgt_dict.unk() not in spelling_idxs + ), f"{word} {spelling} {spelling_idxs}" + self.trie.insert(spelling_idxs, word_idx, score) + self.trie.smear(SmearingMode.MAX) + + self.decoder_opts = LexiconDecoderOptions( + beam_size=cfg.beam, + beam_size_token=cfg.beamsizetoken or len(tgt_dict), + beam_threshold=cfg.beamthreshold, + lm_weight=cfg.lmweight, + word_score=cfg.wordscore, + unk_score=cfg.unkweight, + sil_score=cfg.silweight, + log_add=False, + criterion_type=CriterionType.CTC, + ) + + self.decoder = LexiconDecoder( + self.decoder_opts, + self.trie, + self.lm, + self.silence, + self.blank, + self.unk_word, + [], + self.unitlm, + ) + else: + assert self.unitlm, "Lexicon-free decoding requires unit LM" + + self.word_dict = flDictionary() + for sym in tgt_dict.symbols: + self.word_dict.add_entry(sym, tgt_dict.index(sym)) + self.lm = KenLM(cfg.lmpath, self.word_dict) + self.decoder_opts = LexiconFreeDecoderOptions( + beam_size=cfg.beam, + beam_size_token=cfg.beamsizetoken or len(tgt_dict), + beam_threshold=cfg.beamthreshold, + lm_weight=cfg.lmweight, + sil_score=cfg.silweight, + log_add=False, + criterion_type=CriterionType.CTC, + ) + self.decoder = LexiconFreeDecoder( + self.decoder_opts, self.lm, self.silence, self.blank, [] + ) + + def get_timesteps(self, token_idxs: List[int]) -> List[int]: + """Returns frame numbers corresponding to every non-blank token. + + Parameters + ---------- + token_idxs : List[int] + IDs of decoded tokens. + + Returns + ------- + List[int] + Frame numbers corresponding to every non-blank token. + """ + timesteps = [] + for i, token_idx in enumerate(token_idxs): + if token_idx == self.blank: + continue + if i == 0 or token_idx != token_idxs[i-1]: + timesteps.append(i) + return timesteps + + def decode( + self, + emissions: torch.FloatTensor, + ) -> List[List[Dict[str, torch.LongTensor]]]: + B, T, N = emissions.size() + hypos = [] + for b in range(B): + emissions_ptr = emissions.data_ptr() + 4 * b * emissions.stride(0) + results = self.decoder.decode(emissions_ptr, T, N) + + nbest_results = results[: self.nbest] + hypos.append( + [ + { + "tokens": self.get_tokens(result.tokens), + "score": result.score, + "timesteps": self.get_timesteps(result.tokens), + "words": [ + self.word_dict.get_entry(x) for x in result.words if x >= 0 + ], + } + for result in nbest_results + ] + ) + return hypos + + +FairseqLMState = namedtuple( + "FairseqLMState", + [ + "prefix", + "incremental_state", + "probs", + ], +) + + +class FairseqLM(LM): + def __init__(self, dictionary: Dictionary, model: FairseqModel) -> None: + super().__init__() + + self.dictionary = dictionary + self.model = model + self.unk = self.dictionary.unk() + + self.save_incremental = False # this currently does not work properly + self.max_cache = 20_000 + + if torch.cuda.is_available(): + model.cuda() + model.eval() + model.make_generation_fast_() + + self.states = {} + self.stateq = deque() + + def start(self, start_with_nothing: bool) -> LMState: + state = LMState() + prefix = torch.LongTensor([[self.dictionary.eos()]]) + incremental_state = {} if self.save_incremental else None + with torch.no_grad(): + res = self.model(prefix.cuda(), incremental_state=incremental_state) + probs = self.model.get_normalized_probs(res, log_probs=True, sample=None) + + if incremental_state is not None: + incremental_state = apply_to_sample(lambda x: x.cpu(), incremental_state) + self.states[state] = FairseqLMState( + prefix.numpy(), incremental_state, probs[0, -1].cpu().numpy() + ) + self.stateq.append(state) + + return state + + def score( + self, + state: LMState, + token_index: int, + no_cache: bool = False, + ) -> Tuple[LMState, int]: + """ + Evaluate language model based on the current lm state and new word + Parameters: + ----------- + state: current lm state + token_index: index of the word + (can be lexicon index then you should store inside LM the + mapping between indices of lexicon and lm, or lm index of a word) + Returns: + -------- + (LMState, float): pair of (new state, score for the current word) + """ + curr_state = self.states[state] + + def trim_cache(targ_size: int) -> None: + while len(self.stateq) > targ_size: + rem_k = self.stateq.popleft() + rem_st = self.states[rem_k] + rem_st = FairseqLMState(rem_st.prefix, None, None) + self.states[rem_k] = rem_st + + if curr_state.probs is None: + new_incremental_state = ( + curr_state.incremental_state.copy() + if curr_state.incremental_state is not None + else None + ) + with torch.no_grad(): + if new_incremental_state is not None: + new_incremental_state = apply_to_sample( + lambda x: x.cuda(), new_incremental_state + ) + elif self.save_incremental: + new_incremental_state = {} + + res = self.model( + torch.from_numpy(curr_state.prefix).cuda(), + incremental_state=new_incremental_state, + ) + probs = self.model.get_normalized_probs( + res, log_probs=True, sample=None + ) + + if new_incremental_state is not None: + new_incremental_state = apply_to_sample( + lambda x: x.cpu(), new_incremental_state + ) + + curr_state = FairseqLMState( + curr_state.prefix, new_incremental_state, probs[0, -1].cpu().numpy() + ) + + if not no_cache: + self.states[state] = curr_state + self.stateq.append(state) + + score = curr_state.probs[token_index].item() + + trim_cache(self.max_cache) + + outstate = state.child(token_index) + if outstate not in self.states and not no_cache: + prefix = np.concatenate( + [curr_state.prefix, torch.LongTensor([[token_index]])], -1 + ) + incr_state = curr_state.incremental_state + + self.states[outstate] = FairseqLMState(prefix, incr_state, None) + + if token_index == self.unk: + score = float("-inf") + + return outstate, score + + def finish(self, state: LMState) -> Tuple[LMState, int]: + """ + Evaluate eos for language model based on the current lm state + Returns: + -------- + (LMState, float): pair of (new state, score for the current word) + """ + return self.score(state, self.dictionary.eos()) + + def empty_cache(self) -> None: + self.states = {} + self.stateq = deque() + gc.collect() + + +class FairseqLMDecoder(BaseDecoder): + def __init__(self, cfg: FlashlightDecoderConfig, tgt_dict: Dictionary) -> None: + super().__init__(tgt_dict) + + self.nbest = cfg.nbest + self.unitlm = cfg.unitlm + + self.lexicon = load_words(cfg.lexicon) if cfg.lexicon else None + self.idx_to_wrd = {} + + checkpoint = torch.load(cfg.lmpath, map_location="cpu") + + if "cfg" in checkpoint and checkpoint["cfg"] is not None: + lm_args = checkpoint["cfg"] + else: + lm_args = convert_namespace_to_omegaconf(checkpoint["args"]) + + if not OmegaConf.is_dict(lm_args): + lm_args = OmegaConf.create(lm_args) + + with open_dict(lm_args.task): + lm_args.task.data = osp.dirname(cfg.lmpath) + + task = tasks.setup_task(lm_args.task) + model = task.build_model(lm_args.model) + model.load_state_dict(checkpoint["model"], strict=False) + + self.trie = Trie(self.vocab_size, self.silence) + + self.word_dict = task.dictionary + self.unk_word = self.word_dict.unk() + self.lm = FairseqLM(self.word_dict, model) + + if self.lexicon: + start_state = self.lm.start(False) + for i, (word, spellings) in enumerate(self.lexicon.items()): + if self.unitlm: + word_idx = i + self.idx_to_wrd[i] = word + score = 0 + else: + word_idx = self.word_dict.index(word) + _, score = self.lm.score(start_state, word_idx, no_cache=True) + + for spelling in spellings: + spelling_idxs = [tgt_dict.index(token) for token in spelling] + assert ( + tgt_dict.unk() not in spelling_idxs + ), f"{spelling} {spelling_idxs}" + self.trie.insert(spelling_idxs, word_idx, score) + self.trie.smear(SmearingMode.MAX) + + self.decoder_opts = LexiconDecoderOptions( + beam_size=cfg.beam, + beam_size_token=cfg.beamsizetoken or len(tgt_dict), + beam_threshold=cfg.beamthreshold, + lm_weight=cfg.lmweight, + word_score=cfg.wordscore, + unk_score=cfg.unkweight, + sil_score=cfg.silweight, + log_add=False, + criterion_type=CriterionType.CTC, + ) + + self.decoder = LexiconDecoder( + self.decoder_opts, + self.trie, + self.lm, + self.silence, + self.blank, + self.unk_word, + [], + self.unitlm, + ) + else: + assert self.unitlm, "Lexicon-free decoding requires unit LM" + + d = {w: [[w]] for w in tgt_dict.symbols} + self.word_dict = create_word_dict(d) + self.lm = KenLM(cfg.lmpath, self.word_dict) + self.decoder_opts = LexiconFreeDecoderOptions( + beam_size=cfg.beam, + beam_size_token=cfg.beamsizetoken or len(tgt_dict), + beam_threshold=cfg.beamthreshold, + lm_weight=cfg.lmweight, + sil_score=cfg.silweight, + log_add=False, + criterion_type=CriterionType.CTC, + ) + self.decoder = LexiconFreeDecoder( + self.decoder_opts, self.lm, self.silence, self.blank, [] + ) + + def decode( + self, + emissions: torch.FloatTensor, + ) -> List[List[Dict[str, torch.LongTensor]]]: + B, T, N = emissions.size() + hypos = [] + + def make_hypo(result: DecodeResult) -> Dict[str, Any]: + hypo = { + "tokens": self.get_tokens(result.tokens), + "score": result.score, + } + if self.lexicon: + hypo["words"] = [ + self.idx_to_wrd[x] if self.unitlm else self.word_dict[x] + for x in result.words + if x >= 0 + ] + return hypo + + for b in range(B): + emissions_ptr = emissions.data_ptr() + 4 * b * emissions.stride(0) + results = self.decoder.decode(emissions_ptr, T, N) + + nbest_results = results[: self.nbest] + hypos.append([make_hypo(result) for result in nbest_results]) + self.lm.empty_cache() + + return hypos diff --git a/data/fairseq/examples/speech_recognition/new/decoders/viterbi_decoder.py b/data/fairseq/examples/speech_recognition/new/decoders/viterbi_decoder.py new file mode 100644 index 0000000000000000000000000000000000000000..a35d95e1464291ed27b7c0b46c4936b5a9184503 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/new/decoders/viterbi_decoder.py @@ -0,0 +1,24 @@ +#!/usr/bin/env python3 + +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import torch + +from typing import List, Dict + +from .base_decoder import BaseDecoder + + +class ViterbiDecoder(BaseDecoder): + def decode( + self, + emissions: torch.FloatTensor, + ) -> List[List[Dict[str, torch.LongTensor]]]: + def get_pred(e): + score = e.log_softmax(dim=-1).max(dim=-1)[0].sum() + toks = e.argmax(dim=-1).unique_consecutive() + return {"tokens":toks[toks != self.blank], "score":score} + return [[get_pred(x)] for x in emissions] diff --git a/data/fairseq/examples/speech_recognition/new/infer.py b/data/fairseq/examples/speech_recognition/new/infer.py new file mode 100644 index 0000000000000000000000000000000000000000..ca5cea4a7c20dbc3eb3787b2f44e3e3817ab8f27 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/new/infer.py @@ -0,0 +1,502 @@ +#!/usr/bin/env python -u +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import ast +import hashlib +import logging +import os +import shutil +import sys +import re +from dataclasses import dataclass, field, is_dataclass +from pathlib import Path +from typing import Any, Dict, List, Optional, Tuple, Union + +import editdistance +import torch +import torch.distributed as dist +from examples.speech_recognition.new.decoders.decoder_config import ( + DecoderConfig, + FlashlightDecoderConfig, +) +from examples.speech_recognition.new.decoders.decoder import Decoder +from fairseq import checkpoint_utils, distributed_utils, progress_bar, tasks, utils +from fairseq.data.data_utils import post_process +from fairseq.dataclass.configs import ( + CheckpointConfig, + CommonConfig, + CommonEvalConfig, + DatasetConfig, + DistributedTrainingConfig, + FairseqDataclass, +) +from fairseq.logging.meters import StopwatchMeter, TimeMeter +from fairseq.logging.progress_bar import BaseProgressBar +from fairseq.models.fairseq_model import FairseqModel +from omegaconf import OmegaConf + +import hydra +from hydra.core.config_store import ConfigStore + +logging.root.setLevel(logging.INFO) +logging.basicConfig(level=logging.INFO) +logger = logging.getLogger(__name__) + +config_path = Path(__file__).resolve().parent / "conf" + + +@dataclass +class DecodingConfig(DecoderConfig, FlashlightDecoderConfig): + unique_wer_file: bool = field( + default=False, + metadata={"help": "If set, use a unique file for storing WER"}, + ) + results_path: Optional[str] = field( + default=None, + metadata={ + "help": "If set, write hypothesis and reference sentences into this directory" + }, + ) + + +@dataclass +class InferConfig(FairseqDataclass): + task: Any = None + decoding: DecodingConfig = DecodingConfig() + common: CommonConfig = CommonConfig() + common_eval: CommonEvalConfig = CommonEvalConfig() + checkpoint: CheckpointConfig = CheckpointConfig() + distributed_training: DistributedTrainingConfig = DistributedTrainingConfig() + dataset: DatasetConfig = DatasetConfig() + is_ax: bool = field( + default=False, + metadata={ + "help": "if true, assumes we are using ax for tuning and returns a tuple for ax to consume" + }, + ) + + +def reset_logging(): + root = logging.getLogger() + for handler in root.handlers: + root.removeHandler(handler) + root.setLevel(os.environ.get("LOGLEVEL", "INFO").upper()) + handler = logging.StreamHandler(sys.stdout) + handler.setFormatter( + logging.Formatter( + fmt="%(asctime)s | %(levelname)s | %(name)s | %(message)s", + datefmt="%Y-%m-%d %H:%M:%S", + ) + ) + root.addHandler(handler) + + +class InferenceProcessor: + cfg: InferConfig + + def __init__(self, cfg: InferConfig) -> None: + self.cfg = cfg + self.task = tasks.setup_task(cfg.task) + + models, saved_cfg = self.load_model_ensemble() + + ### LOAD ADAPTER #### + ckpt_obj = checkpoint_utils.load_checkpoint_to_cpu(self.cfg.common_eval.path) + if "adapter" in ckpt_obj: + target_lang = self.cfg.dataset.gen_subset.split(":")[0] + assert target_lang in ckpt_obj["adapter"] + + logger.info(f">>> LOADING ADAPTER: {target_lang}") + ft_obj = ckpt_obj["adapter"][target_lang] + ft_model = ft_obj["model"] + cdevice = models[0].w2v_encoder.proj.weight.device + cdtype = models[0].w2v_encoder.proj.weight.dtype + ft_proj_out, ft_proj_in = ft_model["w2v_encoder.proj.weight"].shape + ft_proj = torch.nn.Linear(ft_proj_in, ft_proj_out, bias=True) + ft_proj.to(device=cdevice, dtype=cdtype) + models[0].w2v_encoder.proj = ft_proj + with torch.no_grad(): + for kk, vv in models[0].named_parameters(): + if kk in ft_model: + vv.copy_(ft_model[kk]) + self.task.load_state_dict(ft_obj["task_state"]) + # overwrite gen_subset with master config + self.cfg.dataset.gen_subset = re.sub('^[\w-]+:', saved_cfg['task']['multi_corpus_keys']+":", self.cfg.dataset.gen_subset) + self.models = models + self.saved_cfg = saved_cfg + self.tgt_dict = self.task.target_dictionary + + self.task.load_dataset( + self.cfg.dataset.gen_subset, + task_cfg=saved_cfg.task, + ) + self.generator = Decoder(cfg.decoding, self.tgt_dict) + self.gen_timer = StopwatchMeter() + self.wps_meter = TimeMeter() + self.num_sentences = 0 + self.total_errors = 0 + self.total_length = 0 + + self.hypo_words_file = None + self.hypo_units_file = None + self.ref_words_file = None + self.ref_units_file = None + self.score_file = None + + self.progress_bar = self.build_progress_bar() + + def __enter__(self) -> "InferenceProcessor": + if self.cfg.decoding.results_path is not None: + self.hypo_words_file = self.get_res_file("hypo.word") + self.hypo_units_file = self.get_res_file("hypo.units") + self.ref_words_file = self.get_res_file("ref.word") + self.ref_units_file = self.get_res_file("ref.units") + self.score_file = self.get_res_file("asr_score") + return self + + def __exit__(self, *exc) -> bool: + if self.cfg.decoding.results_path is not None: + self.hypo_words_file.close() + self.hypo_units_file.close() + self.ref_words_file.close() + self.ref_units_file.close() + self.score_file.close() + return False + + def __iter__(self) -> Any: + for sample in self.progress_bar: + if not self.cfg.common.cpu: + sample = utils.move_to_cuda(sample) + + # Happens on the last batch. + if "net_input" not in sample: + continue + yield sample + + def log(self, *args, **kwargs): + self.progress_bar.log(*args, **kwargs) + + def print(self, *args, **kwargs): + self.progress_bar.print(*args, **kwargs) + + def get_res_file(self, fname: str) -> None: + fname = os.path.join(self.cfg.decoding.results_path, fname) + if self.data_parallel_world_size > 1: + fname = f"{fname}.{self.data_parallel_rank}" + return open(fname, "w", buffering=1) + + def merge_shards(self) -> None: + """Merges all shard files into shard 0, then removes shard suffix.""" + + shard_id = self.data_parallel_rank + num_shards = self.data_parallel_world_size + + if self.data_parallel_world_size > 1: + + def merge_shards_with_root(fname: str) -> None: + fname = os.path.join(self.cfg.decoding.results_path, fname) + logger.info("Merging %s on shard %d", fname, shard_id) + base_fpath = Path(f"{fname}.0") + with open(base_fpath, "a") as out_file: + for s in range(1, num_shards): + shard_fpath = Path(f"{fname}.{s}") + with open(shard_fpath, "r") as in_file: + for line in in_file: + out_file.write(line) + shard_fpath.unlink() + shutil.move(f"{fname}.0", fname) + + dist.barrier() # ensure all shards finished writing + if shard_id == (0 % num_shards): + merge_shards_with_root("hypo.word") + if shard_id == (1 % num_shards): + merge_shards_with_root("hypo.units") + if shard_id == (2 % num_shards): + merge_shards_with_root("ref.word") + if shard_id == (3 % num_shards): + merge_shards_with_root("ref.units") + dist.barrier() + + def optimize_model(self, model: FairseqModel) -> None: + model.make_generation_fast_() + if self.cfg.common.fp16: + model.half() + if not self.cfg.common.cpu: + model.cuda() + + def load_model_ensemble(self) -> Tuple[List[FairseqModel], FairseqDataclass]: + arg_overrides = ast.literal_eval(self.cfg.common_eval.model_overrides) + models, saved_cfg = checkpoint_utils.load_model_ensemble( + utils.split_paths(self.cfg.common_eval.path, separator="\\"), + arg_overrides=arg_overrides, + task=self.task, + suffix=self.cfg.checkpoint.checkpoint_suffix, + strict=(self.cfg.checkpoint.checkpoint_shard_count == 1), + num_shards=self.cfg.checkpoint.checkpoint_shard_count, + ) + for model in models: + self.optimize_model(model) + return models, saved_cfg + + def get_dataset_itr(self, disable_iterator_cache: bool = False) -> None: + return self.task.get_batch_iterator( + dataset=self.task.dataset(self.cfg.dataset.gen_subset), + max_tokens=self.cfg.dataset.max_tokens, + max_sentences=self.cfg.dataset.batch_size, + max_positions=(sys.maxsize, sys.maxsize), + ignore_invalid_inputs=self.cfg.dataset.skip_invalid_size_inputs_valid_test, + required_batch_size_multiple=self.cfg.dataset.required_batch_size_multiple, + seed=self.cfg.common.seed, + num_shards=self.data_parallel_world_size, + shard_id=self.data_parallel_rank, + num_workers=self.cfg.dataset.num_workers, + data_buffer_size=self.cfg.dataset.data_buffer_size, + disable_iterator_cache=disable_iterator_cache, + ).next_epoch_itr(shuffle=False) + + def build_progress_bar( + self, + epoch: Optional[int] = None, + prefix: Optional[str] = None, + default_log_format: str = "tqdm", + ) -> BaseProgressBar: + return progress_bar.progress_bar( + iterator=self.get_dataset_itr(), + log_format=self.cfg.common.log_format, + log_interval=self.cfg.common.log_interval, + epoch=epoch, + prefix=prefix, + tensorboard_logdir=self.cfg.common.tensorboard_logdir, + default_log_format=default_log_format, + ) + + @property + def data_parallel_world_size(self): + if self.cfg.distributed_training.distributed_world_size == 1: + return 1 + return distributed_utils.get_data_parallel_world_size() + + @property + def data_parallel_rank(self): + if self.cfg.distributed_training.distributed_world_size == 1: + return 0 + return distributed_utils.get_data_parallel_rank() + + def process_sentence( + self, + sample: Dict[str, Any], + hypo: Dict[str, Any], + sid: int, + batch_id: int, + ) -> Tuple[int, int]: + speaker = None # Speaker can't be parsed from dataset. + if "target_label" in sample: + toks = sample["target_label"] + else: + toks = sample["target"] + toks = toks[batch_id, :] + + # Processes hypothesis. + hyp_pieces = self.tgt_dict.string(hypo["tokens"].int().cpu()) + if "words" in hypo: + hyp_words = " ".join(hypo["words"]) + else: + hyp_words = post_process(hyp_pieces, self.cfg.common_eval.post_process) + + # Processes target. + target_tokens = utils.strip_pad(toks, self.tgt_dict.pad()) + tgt_pieces = self.tgt_dict.string(target_tokens.int().cpu()) + tgt_words = post_process(tgt_pieces, self.cfg.common_eval.post_process) + + if self.cfg.decoding.results_path is not None: + print(f"{hyp_pieces} ({speaker}-{sid})", file=self.hypo_units_file) + print(f"{hyp_words} ({speaker}-{sid})", file=self.hypo_words_file) + print(f"{tgt_pieces} ({speaker}-{sid})", file=self.ref_units_file) + print(f"{tgt_words} ({speaker}-{sid})", file=self.ref_words_file) + print(f"{hypo['score'].item()} ({speaker}-{sid})", file=self.score_file) + + if not self.cfg.common_eval.quiet: + logger.info(f"HYPO: {hyp_words}") + logger.info(f"REF: {tgt_words}") + logger.info("---------------------") + + hyp_words, tgt_words = hyp_words.split(), tgt_words.split() + + return editdistance.eval(hyp_words, tgt_words), len(tgt_words) + + def process_sample(self, sample: Dict[str, Any]) -> None: + self.gen_timer.start() + hypos = self.task.inference_step( + generator=self.generator, + models=self.models, + sample=sample, + ) + num_generated_tokens = sum(len(h[0]["tokens"]) for h in hypos) + self.gen_timer.stop(num_generated_tokens) + self.wps_meter.update(num_generated_tokens) + + for batch_id, sample_id in enumerate(sample["id"].tolist()): + errs, length = self.process_sentence( + sample=sample, + sid=sample_id, + batch_id=batch_id, + hypo=hypos[batch_id][0], + ) + self.total_errors += errs + self.total_length += length + + self.log({"wps": round(self.wps_meter.avg)}) + if "nsentences" in sample: + self.num_sentences += sample["nsentences"] + else: + self.num_sentences += sample["id"].numel() + + def log_generation_time(self) -> None: + logger.info( + "Processed %d sentences (%d tokens) in %.1fs %.2f " + "sentences per second, %.2f tokens per second)", + self.num_sentences, + self.gen_timer.n, + self.gen_timer.sum, + self.num_sentences / (self.gen_timer.sum + 1e-6), + 1.0 / (self.gen_timer.avg + 1e-6), + ) + + +def parse_wer(wer_file: Path) -> float: + with open(wer_file, "r") as f: + return float(f.readline().strip().split(" ")[1]) + + +def get_wer_file(cfg: InferConfig) -> Path: + """Hashes the decoding parameters to a unique file ID.""" + base_path = "wer" + if cfg.decoding.results_path is not None: + base_path = os.path.join(cfg.decoding.results_path, base_path) + + if cfg.decoding.unique_wer_file: + yaml_str = OmegaConf.to_yaml(cfg.decoding) + fid = int(hashlib.md5(yaml_str.encode("utf-8")).hexdigest(), 16) + return Path(f"{base_path}.{fid % 1000000}") + else: + return Path(base_path) + + +def main(cfg: InferConfig) -> float: + """Entry point for main processing logic. + + Args: + cfg: The inferance configuration to use. + wer: Optional shared memory pointer for returning the WER. If not None, + the final WER value will be written here instead of being returned. + + Returns: + The final WER if `wer` is None, otherwise None. + """ + + yaml_str, wer_file = OmegaConf.to_yaml(cfg.decoding), get_wer_file(cfg) + + # Validates the provided configuration. + if cfg.dataset.max_tokens is None and cfg.dataset.batch_size is None: + cfg.dataset.max_tokens = 4000000 + if not cfg.common.cpu and not torch.cuda.is_available(): + raise ValueError("CUDA not found; set `cpu=True` to run without CUDA") + + logger.info(cfg.common_eval.path) + + with InferenceProcessor(cfg) as processor: + for sample in processor: + processor.process_sample(sample) + + processor.log_generation_time() + + if cfg.decoding.results_path is not None: + processor.merge_shards() + + errs_t, leng_t = processor.total_errors, processor.total_length + + if cfg.common.cpu: + logger.warning("Merging WER requires CUDA.") + elif processor.data_parallel_world_size > 1: + stats = torch.LongTensor([errs_t, leng_t]).cuda() + dist.all_reduce(stats, op=dist.ReduceOp.SUM) + errs_t, leng_t = stats[0].item(), stats[1].item() + + wer = errs_t * 100.0 / leng_t + + if distributed_utils.is_master(cfg.distributed_training): + with open(wer_file, "w") as f: + f.write( + ( + f"WER: {wer}\n" + f"err / num_ref_words = {errs_t} / {leng_t}\n\n" + f"{yaml_str}" + ) + ) + + return wer + + +@hydra.main(config_path=config_path, config_name="infer") +def hydra_main(cfg: InferConfig) -> Union[float, Tuple[float, Optional[float]]]: + container = OmegaConf.to_container(cfg, resolve=True, enum_to_str=True) + cfg = OmegaConf.create(container) + OmegaConf.set_struct(cfg, True) + + if cfg.common.reset_logging: + reset_logging() + + utils.import_user_module(cfg.common) + + # logger.info("Config:\n%s", OmegaConf.to_yaml(cfg)) + wer = float("inf") + + try: + if cfg.common.profile: + with torch.cuda.profiler.profile(): + with torch.autograd.profiler.emit_nvtx(): + distributed_utils.call_main(cfg, main) + else: + distributed_utils.call_main(cfg, main) + + wer = parse_wer(get_wer_file(cfg)) + except BaseException as e: # pylint: disable=broad-except + if not cfg.common.suppress_crashes: + raise + else: + logger.error("Crashed! %s", str(e)) + + logger.info("Word error rate: %.4f", wer) + if cfg.is_ax: + return wer, None + + return wer + + +def cli_main() -> None: + try: + from hydra._internal.utils import ( + get_args, + ) # pylint: disable=import-outside-toplevel + + cfg_name = get_args().config_name or "infer" + except ImportError: + logger.warning("Failed to get config name from hydra args") + cfg_name = "infer" + + cs = ConfigStore.instance() + cs.store(name=cfg_name, node=InferConfig) + + for k in InferConfig.__dataclass_fields__: + if is_dataclass(InferConfig.__dataclass_fields__[k].type): + v = InferConfig.__dataclass_fields__[k].default + cs.store(name=k, node=v) + + hydra_main() # pylint: disable=no-value-for-parameter + + +if __name__ == "__main__": + cli_main() diff --git a/data/fairseq/examples/speech_recognition/tasks/__init__.py b/data/fairseq/examples/speech_recognition/tasks/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..7ac3b8dc69639c92cc129294356e9012745e3fb2 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/tasks/__init__.py @@ -0,0 +1,8 @@ +import importlib +import os + + +for file in sorted(os.listdir(os.path.dirname(__file__))): + if file.endswith(".py") and not file.startswith("_"): + task_name = file[: file.find(".py")] + importlib.import_module("examples.speech_recognition.tasks." + task_name) diff --git a/data/fairseq/examples/speech_recognition/tasks/speech_recognition.py b/data/fairseq/examples/speech_recognition/tasks/speech_recognition.py new file mode 100644 index 0000000000000000000000000000000000000000..d9f011d55ff4fdfeb4c04ca790c314d685708c3a --- /dev/null +++ b/data/fairseq/examples/speech_recognition/tasks/speech_recognition.py @@ -0,0 +1,157 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import json +import os +import re +import sys + +import torch +from examples.speech_recognition.data import AsrDataset +from examples.speech_recognition.data.replabels import replabel_symbol +from fairseq.data import Dictionary +from fairseq.tasks import LegacyFairseqTask, register_task + + +def get_asr_dataset_from_json(data_json_path, tgt_dict): + """ + Parse data json and create dataset. + See scripts/asr_prep_json.py which pack json from raw files + + Json example: + { + "utts": { + "4771-29403-0025": { + "input": { + "length_ms": 170, + "path": "/tmp/file1.flac" + }, + "output": { + "text": "HELLO \n", + "token": "HE LLO", + "tokenid": "4815, 861" + } + }, + "1564-142299-0096": { + ... + } + } + """ + if not os.path.isfile(data_json_path): + raise FileNotFoundError("Dataset not found: {}".format(data_json_path)) + with open(data_json_path, "rb") as f: + data_samples = json.load(f)["utts"] + assert len(data_samples) != 0 + sorted_samples = sorted( + data_samples.items(), + key=lambda sample: int(sample[1]["input"]["length_ms"]), + reverse=True, + ) + aud_paths = [s[1]["input"]["path"] for s in sorted_samples] + ids = [s[0] for s in sorted_samples] + speakers = [] + for s in sorted_samples: + m = re.search("(.+?)-(.+?)-(.+?)", s[0]) + speakers.append(m.group(1) + "_" + m.group(2)) + frame_sizes = [s[1]["input"]["length_ms"] for s in sorted_samples] + tgt = [ + [int(i) for i in s[1]["output"]["tokenid"].split(", ")] + for s in sorted_samples + ] + # append eos + tgt = [[*t, tgt_dict.eos()] for t in tgt] + return AsrDataset(aud_paths, frame_sizes, tgt, tgt_dict, ids, speakers) + + +@register_task("speech_recognition") +class SpeechRecognitionTask(LegacyFairseqTask): + """ + Task for training speech recognition model. + """ + + @staticmethod + def add_args(parser): + """Add task-specific arguments to the parser.""" + parser.add_argument("data", help="path to data directory") + parser.add_argument( + "--silence-token", default="\u2581", help="token for silence (used by w2l)" + ) + parser.add_argument( + "--max-source-positions", + default=sys.maxsize, + type=int, + metavar="N", + help="max number of frames in the source sequence", + ) + parser.add_argument( + "--max-target-positions", + default=1024, + type=int, + metavar="N", + help="max number of tokens in the target sequence", + ) + + def __init__(self, args, tgt_dict): + super().__init__(args) + self.tgt_dict = tgt_dict + + @classmethod + def setup_task(cls, args, **kwargs): + """Setup the task (e.g., load dictionaries).""" + dict_path = os.path.join(args.data, "dict.txt") + if not os.path.isfile(dict_path): + raise FileNotFoundError("Dict not found: {}".format(dict_path)) + tgt_dict = Dictionary.load(dict_path) + + if args.criterion == "ctc_loss": + tgt_dict.add_symbol("") + elif args.criterion == "asg_loss": + for i in range(1, args.max_replabel + 1): + tgt_dict.add_symbol(replabel_symbol(i)) + + print("| dictionary: {} types".format(len(tgt_dict))) + return cls(args, tgt_dict) + + def load_dataset(self, split, combine=False, **kwargs): + """Load a given dataset split. + + Args: + split (str): name of the split (e.g., train, valid, test) + """ + data_json_path = os.path.join(self.args.data, "{}.json".format(split)) + self.datasets[split] = get_asr_dataset_from_json(data_json_path, self.tgt_dict) + + def build_generator(self, models, args, **unused): + w2l_decoder = getattr(args, "w2l_decoder", None) + if w2l_decoder == "viterbi": + from examples.speech_recognition.w2l_decoder import W2lViterbiDecoder + + return W2lViterbiDecoder(args, self.target_dictionary) + elif w2l_decoder == "kenlm": + from examples.speech_recognition.w2l_decoder import W2lKenLMDecoder + + return W2lKenLMDecoder(args, self.target_dictionary) + elif w2l_decoder == "fairseqlm": + from examples.speech_recognition.w2l_decoder import W2lFairseqLMDecoder + + return W2lFairseqLMDecoder(args, self.target_dictionary) + else: + return super().build_generator(models, args) + + @property + def target_dictionary(self): + """Return the :class:`~fairseq.data.Dictionary` for the language + model.""" + return self.tgt_dict + + @property + def source_dictionary(self): + """Return the source :class:`~fairseq.data.Dictionary` (if applicable + for this task).""" + return None + + def max_positions(self): + """Return the max speech and sentence length allowed by the task.""" + return (self.args.max_source_positions, self.args.max_target_positions) diff --git a/data/fairseq/examples/speech_recognition/utils/wer_utils.py b/data/fairseq/examples/speech_recognition/utils/wer_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..cf6f3d09ba41a46ad4d7968fb3c286dd53d15c38 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/utils/wer_utils.py @@ -0,0 +1,381 @@ +#!/usr/bin/env python3 + +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +from __future__ import absolute_import, division, print_function, unicode_literals + +import re +from collections import deque +from enum import Enum + +import numpy as np + + +""" + Utility modules for computation of Word Error Rate, + Alignments, as well as more granular metrics like + deletion, insersion and substitutions. +""" + + +class Code(Enum): + match = 1 + substitution = 2 + insertion = 3 + deletion = 4 + + +class Token(object): + def __init__(self, lbl="", st=np.nan, en=np.nan): + if np.isnan(st): + self.label, self.start, self.end = "", 0.0, 0.0 + else: + self.label, self.start, self.end = lbl, st, en + + +class AlignmentResult(object): + def __init__(self, refs, hyps, codes, score): + self.refs = refs # std::deque + self.hyps = hyps # std::deque + self.codes = codes # std::deque + self.score = score # float + + +def coordinate_to_offset(row, col, ncols): + return int(row * ncols + col) + + +def offset_to_row(offset, ncols): + return int(offset / ncols) + + +def offset_to_col(offset, ncols): + return int(offset % ncols) + + +def trimWhitespace(str): + return re.sub(" +", " ", re.sub(" *$", "", re.sub("^ *", "", str))) + + +def str2toks(str): + pieces = trimWhitespace(str).split(" ") + toks = [] + for p in pieces: + toks.append(Token(p, 0.0, 0.0)) + return toks + + +class EditDistance(object): + def __init__(self, time_mediated): + self.time_mediated_ = time_mediated + self.scores_ = np.nan # Eigen::Matrix + self.backtraces_ = ( + np.nan + ) # Eigen::Matrix backtraces_; + self.confusion_pairs_ = {} + + def cost(self, ref, hyp, code): + if self.time_mediated_: + if code == Code.match: + return abs(ref.start - hyp.start) + abs(ref.end - hyp.end) + elif code == Code.insertion: + return hyp.end - hyp.start + elif code == Code.deletion: + return ref.end - ref.start + else: # substitution + return abs(ref.start - hyp.start) + abs(ref.end - hyp.end) + 0.1 + else: + if code == Code.match: + return 0 + elif code == Code.insertion or code == Code.deletion: + return 3 + else: # substitution + return 4 + + def get_result(self, refs, hyps): + res = AlignmentResult(refs=deque(), hyps=deque(), codes=deque(), score=np.nan) + + num_rows, num_cols = self.scores_.shape + res.score = self.scores_[num_rows - 1, num_cols - 1] + + curr_offset = coordinate_to_offset(num_rows - 1, num_cols - 1, num_cols) + + while curr_offset != 0: + curr_row = offset_to_row(curr_offset, num_cols) + curr_col = offset_to_col(curr_offset, num_cols) + + prev_offset = self.backtraces_[curr_row, curr_col] + + prev_row = offset_to_row(prev_offset, num_cols) + prev_col = offset_to_col(prev_offset, num_cols) + + res.refs.appendleft(curr_row - 1) # Note: this was .push_front() in C++ + res.hyps.appendleft(curr_col - 1) + if curr_row - 1 == prev_row and curr_col == prev_col: + res.codes.appendleft(Code.deletion) + elif curr_row == prev_row and curr_col - 1 == prev_col: + res.codes.appendleft(Code.insertion) + else: + # assert(curr_row - 1 == prev_row and curr_col - 1 == prev_col) + ref_str = refs[res.refs[0]].label + hyp_str = hyps[res.hyps[0]].label + + if ref_str == hyp_str: + res.codes.appendleft(Code.match) + else: + res.codes.appendleft(Code.substitution) + + confusion_pair = "%s -> %s" % (ref_str, hyp_str) + if confusion_pair not in self.confusion_pairs_: + self.confusion_pairs_[confusion_pair] = 1 + else: + self.confusion_pairs_[confusion_pair] += 1 + + curr_offset = prev_offset + + return res + + def align(self, refs, hyps): + if len(refs) == 0 and len(hyps) == 0: + return np.nan + + # NOTE: we're not resetting the values in these matrices because every value + # will be overridden in the loop below. If this assumption doesn't hold, + # be sure to set all entries in self.scores_ and self.backtraces_ to 0. + self.scores_ = np.zeros((len(refs) + 1, len(hyps) + 1)) + self.backtraces_ = np.zeros((len(refs) + 1, len(hyps) + 1)) + + num_rows, num_cols = self.scores_.shape + + for i in range(num_rows): + for j in range(num_cols): + if i == 0 and j == 0: + self.scores_[i, j] = 0.0 + self.backtraces_[i, j] = 0 + continue + + if i == 0: + self.scores_[i, j] = self.scores_[i, j - 1] + self.cost( + None, hyps[j - 1], Code.insertion + ) + self.backtraces_[i, j] = coordinate_to_offset(i, j - 1, num_cols) + continue + + if j == 0: + self.scores_[i, j] = self.scores_[i - 1, j] + self.cost( + refs[i - 1], None, Code.deletion + ) + self.backtraces_[i, j] = coordinate_to_offset(i - 1, j, num_cols) + continue + + # Below here both i and j are greater than 0 + ref = refs[i - 1] + hyp = hyps[j - 1] + best_score = self.scores_[i - 1, j - 1] + ( + self.cost(ref, hyp, Code.match) + if (ref.label == hyp.label) + else self.cost(ref, hyp, Code.substitution) + ) + + prev_row = i - 1 + prev_col = j - 1 + ins = self.scores_[i, j - 1] + self.cost(None, hyp, Code.insertion) + if ins < best_score: + best_score = ins + prev_row = i + prev_col = j - 1 + + delt = self.scores_[i - 1, j] + self.cost(ref, None, Code.deletion) + if delt < best_score: + best_score = delt + prev_row = i - 1 + prev_col = j + + self.scores_[i, j] = best_score + self.backtraces_[i, j] = coordinate_to_offset( + prev_row, prev_col, num_cols + ) + + return self.get_result(refs, hyps) + + +class WERTransformer(object): + def __init__(self, hyp_str, ref_str, verbose=True): + self.ed_ = EditDistance(False) + self.id2oracle_errs_ = {} + self.utts_ = 0 + self.words_ = 0 + self.insertions_ = 0 + self.deletions_ = 0 + self.substitutions_ = 0 + + self.process(["dummy_str", hyp_str, ref_str]) + + if verbose: + print("'%s' vs '%s'" % (hyp_str, ref_str)) + self.report_result() + + def process(self, input): # std::vector&& input + if len(input) < 3: + print( + "Input must be of the form ... , got ", + len(input), + " inputs:", + ) + return None + + # Align + # std::vector hyps; + # std::vector refs; + + hyps = str2toks(input[-2]) + refs = str2toks(input[-1]) + + alignment = self.ed_.align(refs, hyps) + if alignment is None: + print("Alignment is null") + return np.nan + + # Tally errors + ins = 0 + dels = 0 + subs = 0 + for code in alignment.codes: + if code == Code.substitution: + subs += 1 + elif code == Code.insertion: + ins += 1 + elif code == Code.deletion: + dels += 1 + + # Output + row = input + row.append(str(len(refs))) + row.append(str(ins)) + row.append(str(dels)) + row.append(str(subs)) + # print(row) + + # Accumulate + kIdIndex = 0 + kNBestSep = "/" + + pieces = input[kIdIndex].split(kNBestSep) + + if len(pieces) == 0: + print( + "Error splitting ", + input[kIdIndex], + " on '", + kNBestSep, + "', got empty list", + ) + return np.nan + + id = pieces[0] + if id not in self.id2oracle_errs_: + self.utts_ += 1 + self.words_ += len(refs) + self.insertions_ += ins + self.deletions_ += dels + self.substitutions_ += subs + self.id2oracle_errs_[id] = [ins, dels, subs] + else: + curr_err = ins + dels + subs + prev_err = np.sum(self.id2oracle_errs_[id]) + if curr_err < prev_err: + self.id2oracle_errs_[id] = [ins, dels, subs] + + return 0 + + def report_result(self): + # print("---------- Summary ---------------") + if self.words_ == 0: + print("No words counted") + return + + # 1-best + best_wer = ( + 100.0 + * (self.insertions_ + self.deletions_ + self.substitutions_) + / self.words_ + ) + + print( + "\tWER = %0.2f%% (%i utts, %i words, %0.2f%% ins, " + "%0.2f%% dels, %0.2f%% subs)" + % ( + best_wer, + self.utts_, + self.words_, + 100.0 * self.insertions_ / self.words_, + 100.0 * self.deletions_ / self.words_, + 100.0 * self.substitutions_ / self.words_, + ) + ) + + def wer(self): + if self.words_ == 0: + wer = np.nan + else: + wer = ( + 100.0 + * (self.insertions_ + self.deletions_ + self.substitutions_) + / self.words_ + ) + return wer + + def stats(self): + if self.words_ == 0: + stats = {} + else: + wer = ( + 100.0 + * (self.insertions_ + self.deletions_ + self.substitutions_) + / self.words_ + ) + stats = dict( + { + "wer": wer, + "utts": self.utts_, + "numwords": self.words_, + "ins": self.insertions_, + "dels": self.deletions_, + "subs": self.substitutions_, + "confusion_pairs": self.ed_.confusion_pairs_, + } + ) + return stats + + +def calc_wer(hyp_str, ref_str): + t = WERTransformer(hyp_str, ref_str, verbose=0) + return t.wer() + + +def calc_wer_stats(hyp_str, ref_str): + t = WERTransformer(hyp_str, ref_str, verbose=0) + return t.stats() + + +def get_wer_alignment_codes(hyp_str, ref_str): + """ + INPUT: hypothesis string, reference string + OUTPUT: List of alignment codes (intermediate results from WER computation) + """ + t = WERTransformer(hyp_str, ref_str, verbose=0) + return t.ed_.align(str2toks(ref_str), str2toks(hyp_str)).codes + + +def merge_counts(x, y): + # Merge two hashes which have 'counts' as their values + # This can be used for example to merge confusion pair counts + # conf_pairs = merge_counts(conf_pairs, stats['confusion_pairs']) + for k, v in y.items(): + if k not in x: + x[k] = 0 + x[k] += v + return x diff --git a/data/fairseq/examples/speech_recognition/w2l_decoder.py b/data/fairseq/examples/speech_recognition/w2l_decoder.py new file mode 100644 index 0000000000000000000000000000000000000000..fbf2d3524ee40bd0d08b6a9560047d96e49b6045 --- /dev/null +++ b/data/fairseq/examples/speech_recognition/w2l_decoder.py @@ -0,0 +1,486 @@ +#!/usr/bin/env python3 + +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +""" +Flashlight decoders. +""" + +import gc +import itertools as it +import os.path as osp +from typing import List +import warnings +from collections import deque, namedtuple + +import numpy as np +import torch +from examples.speech_recognition.data.replabels import unpack_replabels +from fairseq import tasks +from fairseq.utils import apply_to_sample +from omegaconf import open_dict +from fairseq.dataclass.utils import convert_namespace_to_omegaconf + + +try: + from flashlight.lib.text.dictionary import create_word_dict, load_words + from flashlight.lib.sequence.criterion import CpuViterbiPath, get_data_ptr_as_bytes + from flashlight.lib.text.decoder import ( + CriterionType, + LexiconDecoderOptions, + KenLM, + LM, + LMState, + SmearingMode, + Trie, + LexiconDecoder, + ) +except: + warnings.warn( + "flashlight python bindings are required to use this functionality. Please install from https://github.com/facebookresearch/flashlight/tree/master/bindings/python" + ) + LM = object + LMState = object + + +class W2lDecoder(object): + def __init__(self, args, tgt_dict): + self.tgt_dict = tgt_dict + self.vocab_size = len(tgt_dict) + self.nbest = args.nbest + + # criterion-specific init + self.criterion_type = CriterionType.CTC + self.blank = ( + tgt_dict.index("") + if "" in tgt_dict.indices + else tgt_dict.bos() + ) + if "" in tgt_dict.indices: + self.silence = tgt_dict.index("") + elif "|" in tgt_dict.indices: + self.silence = tgt_dict.index("|") + else: + self.silence = tgt_dict.eos() + self.asg_transitions = None + + def generate(self, models, sample, **unused): + """Generate a batch of inferences.""" + # model.forward normally channels prev_output_tokens into the decoder + # separately, but SequenceGenerator directly calls model.encoder + encoder_input = { + k: v for k, v in sample["net_input"].items() if k != "prev_output_tokens" + } + emissions = self.get_emissions(models, encoder_input) + return self.decode(emissions) + + def get_emissions(self, models, encoder_input): + """Run encoder and normalize emissions""" + model = models[0] + encoder_out = model(**encoder_input) + if hasattr(model, "get_logits"): + emissions = model.get_logits(encoder_out) # no need to normalize emissions + else: + emissions = model.get_normalized_probs(encoder_out, log_probs=True) + return emissions.transpose(0, 1).float().cpu().contiguous() + + def get_tokens(self, idxs): + """Normalize tokens by handling CTC blank, ASG replabels, etc.""" + idxs = (g[0] for g in it.groupby(idxs)) + idxs = filter(lambda x: x != self.blank, idxs) + return torch.LongTensor(list(idxs)) + + +class W2lViterbiDecoder(W2lDecoder): + def __init__(self, args, tgt_dict): + super().__init__(args, tgt_dict) + + def decode(self, emissions): + B, T, N = emissions.size() + hypos = [] + if self.asg_transitions is None: + transitions = torch.FloatTensor(N, N).zero_() + else: + transitions = torch.FloatTensor(self.asg_transitions).view(N, N) + viterbi_path = torch.IntTensor(B, T) + workspace = torch.ByteTensor(CpuViterbiPath.get_workspace_size(B, T, N)) + CpuViterbiPath.compute( + B, + T, + N, + get_data_ptr_as_bytes(emissions), + get_data_ptr_as_bytes(transitions), + get_data_ptr_as_bytes(viterbi_path), + get_data_ptr_as_bytes(workspace), + ) + return [ + [{"tokens": self.get_tokens(viterbi_path[b].tolist()), "score": 0}] + for b in range(B) + ] + + +class W2lKenLMDecoder(W2lDecoder): + def __init__(self, args, tgt_dict): + super().__init__(args, tgt_dict) + + self.unit_lm = getattr(args, "unit_lm", False) + + if args.lexicon: + self.lexicon = load_words(args.lexicon) + self.word_dict = create_word_dict(self.lexicon) + self.unk_word = self.word_dict.get_index("") + + self.lm = KenLM(args.kenlm_model, self.word_dict) + self.trie = Trie(self.vocab_size, self.silence) + + start_state = self.lm.start(False) + for i, (word, spellings) in enumerate(self.lexicon.items()): + word_idx = self.word_dict.get_index(word) + _, score = self.lm.score(start_state, word_idx) + for spelling in spellings: + spelling_idxs = [tgt_dict.index(token) for token in spelling] + assert ( + tgt_dict.unk() not in spelling_idxs + ), f"{spelling} {spelling_idxs}" + self.trie.insert(spelling_idxs, word_idx, score) + self.trie.smear(SmearingMode.MAX) + + self.decoder_opts = LexiconDecoderOptions( + beam_size=args.beam, + beam_size_token=int(getattr(args, "beam_size_token", len(tgt_dict))), + beam_threshold=args.beam_threshold, + lm_weight=args.lm_weight, + word_score=args.word_score, + unk_score=args.unk_weight, + sil_score=args.sil_weight, + log_add=False, + criterion_type=self.criterion_type, + ) + + if self.asg_transitions is None: + N = 768 + # self.asg_transitions = torch.FloatTensor(N, N).zero_() + self.asg_transitions = [] + + self.decoder = LexiconDecoder( + self.decoder_opts, + self.trie, + self.lm, + self.silence, + self.blank, + self.unk_word, + self.asg_transitions, + self.unit_lm, + ) + else: + assert args.unit_lm, "lexicon free decoding can only be done with a unit language model" + from flashlight.lib.text.decoder import LexiconFreeDecoder, LexiconFreeDecoderOptions + + d = {w: [[w]] for w in tgt_dict.symbols} + self.word_dict = create_word_dict(d) + self.lm = KenLM(args.kenlm_model, self.word_dict) + self.decoder_opts = LexiconFreeDecoderOptions( + beam_size=args.beam, + beam_size_token=int(getattr(args, "beam_size_token", len(tgt_dict))), + beam_threshold=args.beam_threshold, + lm_weight=args.lm_weight, + sil_score=args.sil_weight, + log_add=False, + criterion_type=self.criterion_type, + ) + self.decoder = LexiconFreeDecoder( + self.decoder_opts, self.lm, self.silence, self.blank, [] + ) + + def get_timesteps(self, token_idxs: List[int]) -> List[int]: + """Returns frame numbers corresponding to every non-blank token. + + Parameters + ---------- + token_idxs : List[int] + IDs of decoded tokens. + + Returns + ------- + List[int] + Frame numbers corresponding to every non-blank token. + """ + timesteps = [] + for i, token_idx in enumerate(token_idxs): + if token_idx == self.blank: + continue + if i == 0 or token_idx != token_idxs[i-1]: + timesteps.append(i) + return timesteps + + def decode(self, emissions): + B, T, N = emissions.size() + hypos = [] + for b in range(B): + emissions_ptr = emissions.data_ptr() + 4 * b * emissions.stride(0) + results = self.decoder.decode(emissions_ptr, T, N) + + nbest_results = results[: self.nbest] + hypos.append( + [ + { + "tokens": self.get_tokens(result.tokens), + "score": result.score, + "timesteps": self.get_timesteps(result.tokens), + "words": [ + self.word_dict.get_entry(x) for x in result.words if x >= 0 + ], + } + for result in nbest_results + ] + ) + return hypos + + +FairseqLMState = namedtuple("FairseqLMState", ["prefix", "incremental_state", "probs"]) + + +class FairseqLM(LM): + def __init__(self, dictionary, model): + LM.__init__(self) + self.dictionary = dictionary + self.model = model + self.unk = self.dictionary.unk() + + self.save_incremental = False # this currently does not work properly + self.max_cache = 20_000 + + model.cuda() + model.eval() + model.make_generation_fast_() + + self.states = {} + self.stateq = deque() + + def start(self, start_with_nothing): + state = LMState() + prefix = torch.LongTensor([[self.dictionary.eos()]]) + incremental_state = {} if self.save_incremental else None + with torch.no_grad(): + res = self.model(prefix.cuda(), incremental_state=incremental_state) + probs = self.model.get_normalized_probs(res, log_probs=True, sample=None) + + if incremental_state is not None: + incremental_state = apply_to_sample(lambda x: x.cpu(), incremental_state) + self.states[state] = FairseqLMState( + prefix.numpy(), incremental_state, probs[0, -1].cpu().numpy() + ) + self.stateq.append(state) + + return state + + def score(self, state: LMState, token_index: int, no_cache: bool = False): + """ + Evaluate language model based on the current lm state and new word + Parameters: + ----------- + state: current lm state + token_index: index of the word + (can be lexicon index then you should store inside LM the + mapping between indices of lexicon and lm, or lm index of a word) + + Returns: + -------- + (LMState, float): pair of (new state, score for the current word) + """ + curr_state = self.states[state] + + def trim_cache(targ_size): + while len(self.stateq) > targ_size: + rem_k = self.stateq.popleft() + rem_st = self.states[rem_k] + rem_st = FairseqLMState(rem_st.prefix, None, None) + self.states[rem_k] = rem_st + + if curr_state.probs is None: + new_incremental_state = ( + curr_state.incremental_state.copy() + if curr_state.incremental_state is not None + else None + ) + with torch.no_grad(): + if new_incremental_state is not None: + new_incremental_state = apply_to_sample( + lambda x: x.cuda(), new_incremental_state + ) + elif self.save_incremental: + new_incremental_state = {} + + res = self.model( + torch.from_numpy(curr_state.prefix).cuda(), + incremental_state=new_incremental_state, + ) + probs = self.model.get_normalized_probs( + res, log_probs=True, sample=None + ) + + if new_incremental_state is not None: + new_incremental_state = apply_to_sample( + lambda x: x.cpu(), new_incremental_state + ) + + curr_state = FairseqLMState( + curr_state.prefix, new_incremental_state, probs[0, -1].cpu().numpy() + ) + + if not no_cache: + self.states[state] = curr_state + self.stateq.append(state) + + score = curr_state.probs[token_index].item() + + trim_cache(self.max_cache) + + outstate = state.child(token_index) + if outstate not in self.states and not no_cache: + prefix = np.concatenate( + [curr_state.prefix, torch.LongTensor([[token_index]])], -1 + ) + incr_state = curr_state.incremental_state + + self.states[outstate] = FairseqLMState(prefix, incr_state, None) + + if token_index == self.unk: + score = float("-inf") + + return outstate, score + + def finish(self, state: LMState): + """ + Evaluate eos for language model based on the current lm state + + Returns: + -------- + (LMState, float): pair of (new state, score for the current word) + """ + return self.score(state, self.dictionary.eos()) + + def empty_cache(self): + self.states = {} + self.stateq = deque() + gc.collect() + + +class W2lFairseqLMDecoder(W2lDecoder): + def __init__(self, args, tgt_dict): + super().__init__(args, tgt_dict) + + self.unit_lm = getattr(args, "unit_lm", False) + + self.lexicon = load_words(args.lexicon) if args.lexicon else None + self.idx_to_wrd = {} + + checkpoint = torch.load(args.kenlm_model, map_location="cpu") + + if "cfg" in checkpoint and checkpoint["cfg"] is not None: + lm_args = checkpoint["cfg"] + else: + lm_args = convert_namespace_to_omegaconf(checkpoint["args"]) + + with open_dict(lm_args.task): + lm_args.task.data = osp.dirname(args.kenlm_model) + + task = tasks.setup_task(lm_args.task) + model = task.build_model(lm_args.model) + model.load_state_dict(checkpoint["model"], strict=False) + + self.trie = Trie(self.vocab_size, self.silence) + + self.word_dict = task.dictionary + self.unk_word = self.word_dict.unk() + self.lm = FairseqLM(self.word_dict, model) + + if self.lexicon: + start_state = self.lm.start(False) + for i, (word, spellings) in enumerate(self.lexicon.items()): + if self.unit_lm: + word_idx = i + self.idx_to_wrd[i] = word + score = 0 + else: + word_idx = self.word_dict.index(word) + _, score = self.lm.score(start_state, word_idx, no_cache=True) + + for spelling in spellings: + spelling_idxs = [tgt_dict.index(token) for token in spelling] + assert ( + tgt_dict.unk() not in spelling_idxs + ), f"{spelling} {spelling_idxs}" + self.trie.insert(spelling_idxs, word_idx, score) + self.trie.smear(SmearingMode.MAX) + + self.decoder_opts = LexiconDecoderOptions( + beam_size=args.beam, + beam_size_token=int(getattr(args, "beam_size_token", len(tgt_dict))), + beam_threshold=args.beam_threshold, + lm_weight=args.lm_weight, + word_score=args.word_score, + unk_score=args.unk_weight, + sil_score=args.sil_weight, + log_add=False, + criterion_type=self.criterion_type, + ) + + self.decoder = LexiconDecoder( + self.decoder_opts, + self.trie, + self.lm, + self.silence, + self.blank, + self.unk_word, + [], + self.unit_lm, + ) + else: + assert args.unit_lm, "lexicon free decoding can only be done with a unit language model" + from flashlight.lib.text.decoder import LexiconFreeDecoder, LexiconFreeDecoderOptions + + d = {w: [[w]] for w in tgt_dict.symbols} + self.word_dict = create_word_dict(d) + self.lm = KenLM(args.kenlm_model, self.word_dict) + self.decoder_opts = LexiconFreeDecoderOptions( + beam_size=args.beam, + beam_size_token=int(getattr(args, "beam_size_token", len(tgt_dict))), + beam_threshold=args.beam_threshold, + lm_weight=args.lm_weight, + sil_score=args.sil_weight, + log_add=False, + criterion_type=self.criterion_type, + ) + self.decoder = LexiconFreeDecoder( + self.decoder_opts, self.lm, self.silence, self.blank, [] + ) + + def decode(self, emissions): + B, T, N = emissions.size() + hypos = [] + + def idx_to_word(idx): + if self.unit_lm: + return self.idx_to_wrd[idx] + else: + return self.word_dict[idx] + + def make_hypo(result): + hypo = {"tokens": self.get_tokens(result.tokens), "score": result.score} + if self.lexicon: + hypo["words"] = [idx_to_word(x) for x in result.words if x >= 0] + return hypo + + for b in range(B): + emissions_ptr = emissions.data_ptr() + 4 * b * emissions.stride(0) + results = self.decoder.decode(emissions_ptr, T, N) + + nbest_results = results[: self.nbest] + hypos.append([make_hypo(result) for result in nbest_results]) + self.lm.empty_cache() + + return hypos diff --git a/data/fairseq/examples/speech_to_speech/README.md b/data/fairseq/examples/speech_to_speech/README.md new file mode 100644 index 0000000000000000000000000000000000000000..f03f6a32f8b3a0070440918b775f18e712480885 --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/README.md @@ -0,0 +1,7 @@ +# Speech to speech translation (S2ST) + +We provide the implementation and resources for the following work on speech-to-speech translation (S2ST): + +* [Direct speech-to-speech translation with discrete units (Lee et al. 2021)](docs/direct_s2st_discrete_units.md) +* [Textless Speech-to-Speech Translation on Real Data (Lee et al. 2021)](docs/textless_s2st_real_data.md) +* [Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation](docs/enhanced_direct_s2st_discrete_units.md) diff --git a/data/fairseq/examples/speech_to_speech/__init__.py b/data/fairseq/examples/speech_to_speech/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..812b3c30b9d0af9616b6e7e75a334bb90934f532 --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/__init__.py @@ -0,0 +1,6 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +from . import unity # noqa diff --git a/data/fairseq/examples/speech_to_speech/asr_bleu/README.md b/data/fairseq/examples/speech_to_speech/asr_bleu/README.md new file mode 100644 index 0000000000000000000000000000000000000000..6a7ea7fcefb1d4dacd02ce726a0a654e09bbc940 --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/asr_bleu/README.md @@ -0,0 +1,34 @@ +# ASR-BLEU evaluation toolkit + +This toolkit provides a set of public ASR models used for evaluation of different speech-to-speech translation systems at FAIR. It enables easier score comparisons between different system's outputs. + +The ASRGenerator wraps different CTC-based ASR models from HuggingFace and fairseq code bases. Torchaudio CTC decoder is built on top of it to decode given audio files. + +Please see `asr_model_cfgs.json` for a list of languages covered currently. + +The high-level pipeline is simple by design: given a lang tag, script loads the ASR model, transcribes model's predicted audio, and computes the BLEU score against provided reference translations using sacrebleu. + +# Dependencies + +Please see `requirements.txt`. + +# Usage examples + +This toolkit have been used with: + +* Speechmatrix project: https://github.com/facebookresearch/fairseq/tree/ust/examples/speech_matrix. + +* Hokkien speech-to-speech translation project: https://github.com/facebookresearch/fairseq/tree/ust/examples/hokkien. + +# Standalone run example + +High-level example, please substitute arguments per your case: + +```bash +python compute_asr_bleu.py --lang \ +--audio_dirpath \ +--reference_path \ +--reference_format txt +``` + +For more details about arguments please see the script argparser help. diff --git a/data/fairseq/examples/speech_to_speech/asr_bleu/__init__.py b/data/fairseq/examples/speech_to_speech/asr_bleu/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/data/fairseq/examples/speech_to_speech/asr_bleu/asr_model_cfgs.json b/data/fairseq/examples/speech_to_speech/asr_bleu/asr_model_cfgs.json new file mode 100644 index 0000000000000000000000000000000000000000..d0a5f3e3aa612b8eb879d9de19ad0d2a7e047fa8 --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/asr_bleu/asr_model_cfgs.json @@ -0,0 +1,198 @@ +{ + "en": { + "oct22": { + "desc": "Wav2Vec 2.0 Large (LV-60) + Self Training from https://github.com/facebookresearch/fairseq/tree/main/examples/wav2vec#pre-trained-models", + "ckpt_path": "https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_vox_960h_pl.pt", + "dict_path": "https://dl.fbaipublicfiles.com/fairseq/wav2vec/dict.ltr.txt", + "model_type": "fairseq", + "lang": "en", + "post_process": "collapse" + } + }, + "hok": { + "oct22": { + "desc": "Hokkien ASR model, for details check [TODO add paper link]", + "ckpt_path": "https://dl.fbaipublicfiles.com/ust_asr/hok/checkpoint_best.pt", + "dict_path": "https://dl.fbaipublicfiles.com/ust_asr/hok/dict.ltr.txt", + "model_type": "fairseq", + "lang": "hok", + "post_process": "none" + } + }, + "es": { + "oct22": { + "model_path": "jonatasgrosman/wav2vec2-large-xlsr-53-spanish", + "model_type": "hf", + "lang": "es", + "post_process": "collapse" + } + }, + "fr": { + "oct22": { + "model_path": "jonatasgrosman/wav2vec2-large-fr-voxpopuli-french", + "model_type": "hf", + "lang": "fr", + "post_process": "collapse" + } + }, + "zh": { + "oct22": { + "model_path": "ydshieh/wav2vec2-large-xlsr-53-chinese-zh-cn-gpt", + "model_type": "hf", + "lang": "zh", + "post_process": "collapse" + } + }, + "tr": { + "oct22": { + "model_path": "cahya/wav2vec2-large-xlsr-turkish-artificial-cv", + "model_type": "hf", + "lang": "tr", + "post_process": "collapse" + } + }, + "ar": { + "oct22": { + "model_path": "jonatasgrosman/wav2vec2-large-xlsr-53-arabic", + "model_type": "hf", + "lang": "ar", + "post_process": "collapse" + } + }, + "vi": { + "oct22": { + "model_path": "not-tanh/wav2vec2-large-xlsr-53-vietnamese", + "model_type": "hf", + "lang": "vi", + "post_process": "collapse" + } + }, + "de": { + "oct22": { + "model_path": "jonatasgrosman/wav2vec2-xls-r-1b-german", + "model_type": "hf", + "lang": "de", + "post_process": "collapse" + } + }, + "pl": { + "oct22": { + "model_path": "jonatasgrosman/wav2vec2-xls-r-1b-polish", + "model_type": "hf", + "lang": "pl", + "post_process": "collapse" + } + }, + "it": { + "oct22": { + "model_path": "jonatasgrosman/wav2vec2-large-xlsr-53-italian", + "model_type": "hf", + "lang": "it", + "post_process": "collapse" + } + }, + "pt": { + "oct22": { + "model_path": "jonatasgrosman/wav2vec2-xls-r-1b-portuguese", + "model_type": "hf", + "lang": "pt", + "post_process": "collapse" + } + }, + "ro": { + "oct22": { + "model_path": "gigant/romanian-wav2vec2", + "model_type": "hf", + "lang": "ro", + "post_process": "collapse" + } + }, + "cs": { + "oct22": { + "model_path": "comodoro/wav2vec2-xls-r-300m-cs-250", + "model_type": "hf", + "lang": "cs", + "post_process": "collapse" + } + }, + "sk": { + "oct22": { + "model_path": "anuragshas/wav2vec2-xls-r-300m-sk-cv8-with-lm", + "model_type": "hf", + "lang": "sk", + "post_process": "collapse" + } + }, + "sl": { + "oct22": { + "model_path": "anuragshas/wav2vec2-xls-r-300m-sl-cv8-with-lm", + "model_type": "hf", + "lang": "sl", + "post_process": "collapse" + } + }, + "fi": { + "oct22": { + "model_path": "jonatasgrosman/wav2vec2-large-xlsr-53-finnish", + "model_type": "hf", + "lang": "fi", + "post_process": "collapse" + } + }, + "hu": { + "oct22": { + "model_path": "jonatasgrosman/wav2vec2-large-xlsr-53-hungarian", + "model_type": "hf", + "lang": "hu", + "post_process": "collapse" + } + }, + "et": { + "oct22": { + "model_path": "RASMUS/wav2vec2-xlsr-1b-et", + "model_type": "hf", + "lang": "et", + "post_process": "collapse" + } + }, + "lt": { + "oct22": { + "model_path": "sammy786/wav2vec2-xlsr-lithuanian", + "model_type": "hf", + "lang": "lt", + "post_process": "collapse" + } + }, + "nl": { + "oct22": { + "model_path": "jonatasgrosman/wav2vec2-xls-r-1b-dutch", + "model_type": "hf", + "lang": "nl", + "post_process": "collapse" + } + }, + "lv": { + "oct22": { + "model_path": "reach-vb/wav2vec2-large-xls-r-1B-common_voice7-lv-ft", + "model_type": "hf", + "lang": "lv", + "post_process": "collapse" + } + }, + "sv": { + "oct22": { + "model_path": "marinone94/xls-r-300m-sv-robust", + "model_type": "hf", + "lang": "sv", + "post_process": "collapse" + } + }, + "hr": { + "oct22": { + "model_path": "classla/wav2vec2-xls-r-parlaspeech-hr", + "model_type": "hf", + "lang": "hr", + "post_process": "collapse" + } + } +} diff --git a/data/fairseq/examples/speech_to_speech/asr_bleu/compute_asr_bleu.py b/data/fairseq/examples/speech_to_speech/asr_bleu/compute_asr_bleu.py new file mode 100644 index 0000000000000000000000000000000000000000..d5926194c186c18c66fe5063a38c5dfa07c989ab --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/asr_bleu/compute_asr_bleu.py @@ -0,0 +1,244 @@ +import os +from typing import Dict, List +import sacrebleu +import pandas as pd +from glob import glob +from pathlib import Path +from utils import retrieve_asr_config, ASRGenerator +from tqdm import tqdm +from argparse import ArgumentParser + + +def merge_tailo_init_final(text): + """ + Hokkien ASR hypothesis post-processing. + """ + sps = text.strip().split() + results = [] + last_syllable = "" + for sp in sps: + if sp == "NULLINIT" or sp == "nullinit": + continue + last_syllable += sp + if sp[-1].isnumeric(): + results.append(last_syllable) + last_syllable = "" + if last_syllable != "": + results.append(last_syllable) + return " ".join(results) + + +def remove_tone(text): + """ + Used for tone-less evaluation of Hokkien + """ + return " ".join([t[:-1] for t in text.split()]) + + +def extract_audio_for_eval(audio_dirpath: str, audio_format: str): + if audio_format == "n_pred.wav": + """ + The assumption here is that 0_pred.wav corresponds to the reference at line position 0 from the reference manifest + """ + audio_list = [] + audio_fp_list = glob((Path(audio_dirpath) / "*_pred.wav").as_posix()) + audio_fp_list = sorted( + audio_fp_list, key=lambda x: int(os.path.basename(x).split("_")[0]) + ) + for i in range(len(audio_fp_list)): + try: + audio_fp = (Path(audio_dirpath) / f"{i}_pred.wav").as_posix() + assert ( + audio_fp in audio_fp_list + ), f"{Path(audio_fp).name} does not exist in {audio_dirpath}" + except AssertionError: + # check the audio with random speaker + audio_fp = Path(audio_dirpath) / f"{i}_spk*_pred.wav" + audio_fp = glob( + audio_fp.as_posix() + ) # resolve audio filepath with random speaker + assert len(audio_fp) == 1 + audio_fp = audio_fp[0] + + audio_list.append(audio_fp) + else: + raise NotImplementedError + + return audio_list + + +def extract_text_for_eval( + references_filepath: str, reference_format: str, reference_tsv_column: str = None +): + if reference_format == "txt": + reference_sentences = open(references_filepath, "r").readlines() + reference_sentences = [l.strip() for l in reference_sentences] + elif reference_format == "tsv": + tsv_df = pd.read_csv(references_filepath, sep="\t", quoting=3) + reference_sentences = tsv_df[reference_tsv_column].to_list() + reference_sentences = [l.strip() for l in reference_sentences] + else: + raise NotImplementedError + + return reference_sentences + + +def compose_eval_data( + audio_dirpath: str, + audio_format: str, + references_filepath: str, + reference_format: str, + reference_tsv_column: str = None, + save_manifest_filepath=None, +): + """ + Speech matrix decoding pipeline produces audio with the following mask "N_pred.wav" where N is the order of the corresponding input sample + """ + + reference_sentences = extract_text_for_eval( + references_filepath, reference_format, reference_tsv_column + ) + predicted_audio_fp_list = extract_audio_for_eval(audio_dirpath, audio_format) + assert len(predicted_audio_fp_list) == len(reference_sentences) + + audio_text_pairs = [ + (audio, reference) + for audio, reference in zip(predicted_audio_fp_list, reference_sentences) + ] + + tsv_manifest = pd.DataFrame(audio_text_pairs, columns=["prediction", "reference"]) + + if save_manifest_filepath is not None: + tsv_manifest.to_csv(save_manifest_filepath, sep="\t", quoting=3) + + return tsv_manifest + + +def load_eval_data_from_tsv(eval_data_filepath: str): + """ + We may load the result of `compose_eval_data` directly if needed + """ + eval_df = pd.from_csv(eval_data_filepath, sep="\t") + + return eval_df + + +def run_asr_bleu(args): + + asr_config = retrieve_asr_config( + args.lang, args.asr_version, json_path="./asr_model_cfgs.json" + ) + asr_model = ASRGenerator(asr_config) + + eval_manifest = compose_eval_data( + audio_dirpath=args.audio_dirpath, + audio_format=args.audio_format, + references_filepath=args.reference_path, + reference_format=args.reference_format, + reference_tsv_column=args.reference_tsv_column, + save_manifest_filepath=None, + ) + + prediction_transcripts = [] + for _, eval_pair in tqdm( + eval_manifest.iterrows(), + desc="Transcribing predictions", + total=len(eval_manifest), + ): + transcription = asr_model.transcribe_audiofile(eval_pair.prediction) + prediction_transcripts.append(transcription.lower()) + + if args.lang == "hok": + prediction_transcripts = [ + merge_tailo_init_final(text) for text in prediction_transcripts + ] + + references = eval_manifest["reference"].tolist() + bleu_score = sacrebleu.corpus_bleu(prediction_transcripts, [references]) + + print(bleu_score) + + return prediction_transcripts, bleu_score + + +def main(): + parser = ArgumentParser( + description="This script computes the ASR-BLEU metric between model's generated audio and the text reference sequences." + ) + + parser.add_argument( + "--lang", + help="The target language used to initialize ASR model, see asr_model_cfgs.json for available languages", + type=str, + ) + parser.add_argument( + "--asr_version", + type=str, + default="oct22", + help="For future support we add and extra layer of asr versions. The current most recent version is oct22 meaning October 2022", + ) + parser.add_argument( + "--audio_dirpath", + type=str, + help="Path to the directory containing the audio predictions from the translation model", + ) + parser.add_argument( + "--reference_path", + type=str, + help="Path to the file containing reference translations in the form of normalized text (to be compared to ASR predictions", + ) + parser.add_argument( + "--reference_format", + choices=["txt", "tsv"], + help="Format of reference file. Txt means plain text format where each line represents single reference sequence", + ) + parser.add_argument( + "--reference_tsv_column", + default=None, + type=str, + help="If format is tsv, then specify the column name which contains reference sequence", + ) + parser.add_argument( + "--audio_format", + default="n_pred.wav", + choices=["n_pred.wav"], + help="Audio format n_pred.wav corresponds to names like 94_pred.wav or 94_spk7_pred.wav where spk7 is the speaker id", + ) + parser.add_argument( + "--results_dirpath", + default=None, + type=str, + help="If specified, the resulting BLEU score will be written to this file path as txt file", + ) + parser.add_argument( + "--transcripts_path", + default=None, + type=str, + help="If specified, the predicted transcripts will be written to this path as a txt file.", + ) + + args = parser.parse_args() + + prediction_transcripts, bleu_score = run_asr_bleu(args) + result_filename = f"{args.reference_format}_{args.lang}_bleu.txt" + if args.results_dirpath is not None: + if not Path(args.results_dirpath).exists(): + Path(args.results_dirpath).mkdir(parents=True) + with open(Path(args.results_dirpath) / result_filename, "w") as f: + f.write(bleu_score.format(width=2)) + + if args.transcripts_path is not None: + with open(args.transcripts_path, "w") as f: + for transcript in prediction_transcripts: + f.write(transcript + "\n") + + +if __name__ == "__main__": + main() + + +""" +Example to load Sl audio and references, compute BLEU: + +export lang=fi; split=vp && python compute_asr_bleu.py --lang $lang --audio_dirpath /checkpoint/hygong/S2S/speech_matrix_release_ckpts/generated_waveform_release/en-$lang/test_$split/checkpoint.pt --audio_format n_pred.wav --reference_path /large_experiments/ust/hygong/S2S/SpeechEncoder/manifests/vp-vp/en-$lang/test_$split.$lang --reference_format txt --results_dirpath ./ +""" diff --git a/data/fairseq/examples/speech_to_speech/asr_bleu/requirements.txt b/data/fairseq/examples/speech_to_speech/asr_bleu/requirements.txt new file mode 100644 index 0000000000000000000000000000000000000000..cfa90f6aeffd879c1e309ae58a2f050c943e87b6 --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/asr_bleu/requirements.txt @@ -0,0 +1,7 @@ +fairseq==0.12.2 +pandas==1.4.3 +sacrebleu==2.2.0 +torch==1.12.1 +torchaudio==0.12.1 +tqdm==4.64.0 +transformers==4.21.1 diff --git a/data/fairseq/examples/speech_to_speech/asr_bleu/utils.py b/data/fairseq/examples/speech_to_speech/asr_bleu/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..0fed55a9b9032952e8cc20db36c127b8354f0478 --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/asr_bleu/utils.py @@ -0,0 +1,306 @@ +import json +import re +import urllib.request +from pathlib import Path + +import fairseq +import torch +from fairseq.data.data_utils import lengths_to_padding_mask +from tqdm import tqdm + +try: + import torchaudio + from torchaudio.models.decoder import ctc_decoder +except ImportError: + raise ImportError("Upgrade torchaudio to 0.12 to enable CTC decoding") + + +class DownloadProgressBar(tqdm): + """A class to represent a download progress bar""" + + def update_to(self, b=1, bsize=1, tsize=None) -> None: + """ + Update the download progress + """ + if tsize is not None: + self.total = tsize + self.update(b * bsize - self.n) + + +def retrieve_asr_config(lang_key: str, asr_version: str, json_path: str) -> dict: + """ + Retrieve the asr model configs + + Args: + lang_key: the lanuage type as the key name + json_path: the path of the config json file + + Returns: + Dict of all the configs in the json file + """ + + with open(json_path, "r") as f: + asr_model_cfgs = json.load(f) + return asr_model_cfgs[lang_key][asr_version] + + +class ASRGenerator(object): + """A class to represent a ASR generator""" + + def __init__( + self, + model_cfg: dict, + cache_dirpath: str = (Path.home() / ".cache" / "ust_asr").as_posix(), + ) -> None: + """ + Construct all the necessary attributes of the ASRGenerator class + + Args: + model_cfg: the dict of the asr model config + cache_dirpath: the default cache path is "Path.home()/.cache/ust_asr" + """ + + self.cache_dirpath = Path(cache_dirpath) / model_cfg["lang"] + self.model_cfg = model_cfg + + self.use_cuda = torch.cuda.is_available() + + torchaudio.set_audio_backend("sox_io") + + if self.model_cfg["model_type"] == "hf": + self.prepare_hf_model(self.model_cfg) + elif self.model_cfg["model_type"] == "fairseq": + self.prepare_fairseq_model(self.model_cfg) + else: + raise NotImplementedError( + f"Model type {self.model_cfg['model_type']} is not supported" + ) + + if self.model_cfg["post_process"] == "collapse": + self.post_process_fn = lambda hypo: "".join(hypo).replace( + self.sil_token, " " + ) + elif self.model_cfg["post_process"] == "none": + self.post_process_fn = lambda hypo: " ".join(hypo).replace( + self.sil_token, " " + ) + else: + raise NotImplementedError + + if self.use_cuda: + self.model.cuda() + self.model.eval() + + self.decoder = ctc_decoder( + lexicon=None, + tokens=self.tokens, + lm=None, + nbest=1, + beam_size=1, + beam_size_token=None, + lm_weight=0.0, + word_score=0.0, + unk_score=float("-inf"), + sil_token=self.sil_token, + sil_score=0.0, + log_add=False, + blank_token=self.blank_token, + ) + + def prepare_hf_model(self, model_cfg: dict) -> None: + """ + Prepare the huggingface asr model + + Args: + model_cfg: dict with the relevant ASR config + """ + + def infer_silence_token(vocab: list): + """ + Different HF checkpoints have different notion of silence token + such as | or " " (space) + Important: when adding new HF asr model in, check what silence token it uses + """ + if "|" in vocab: + return "|" + elif " " in vocab: + return " " + else: + raise RuntimeError("Silence token is not found in the vocabulary") + + try: + from transformers import (AutoFeatureExtractor, AutoTokenizer, + Wav2Vec2ForCTC, Wav2Vec2Processor) + except ImportError: + raise ImportError("Install transformers to load HF wav2vec model") + + model_path = model_cfg["model_path"] + self.model = Wav2Vec2ForCTC.from_pretrained(model_path) + self.tokenizer = AutoTokenizer.from_pretrained(model_path) + self.preprocessor = AutoFeatureExtractor.from_pretrained(model_path) + self.processor = Wav2Vec2Processor.from_pretrained(model_path) + + # extra unk tokens are there to make some models work e.g. Finnish ASR has some vocab issue + vocab_list = [ + self.tokenizer.decoder.get(i, f"{self.tokenizer.unk_token}1") + for i in range(self.tokenizer.vocab_size) + ] + + self.sampling_rate = self.preprocessor.sampling_rate + self.normalize_input = self.preprocessor.do_normalize + self.tokens = vocab_list + self.sil_token = infer_silence_token(vocab_list) + self.blank_token = self.tokenizer.pad_token + + def prepare_fairseq_model(self, model_cfg: dict) -> None: + """ + Prepare the fairseq asr model + + Args: + model_cfg: the specific model config dict must have: (1) ckpt_path, (2) dict_path + """ + + def download_file(url: str, cache_dir: Path): + download_path = cache_dir / url.split("/")[-1] + if not (cache_dir / url.split("/")[-1]).exists(): + with DownloadProgressBar( + unit="B", unit_scale=True, miniters=1, desc=url.split("/")[-1] + ) as t: + cache_dir.mkdir(parents=True, exist_ok=True) + urllib.request.urlretrieve( + url, filename=download_path.as_posix(), reporthook=t.update_to + ) + else: + print(f"'{url}' exists in {cache_dir}") + + return download_path.as_posix() + + try: + ckpt_path = model_cfg["ckpt_path"] + dict_path = model_cfg["dict_path"] + except KeyError: + raise KeyError( + "Fairseq model cfg must provide (1) ckpt_path, (2) dict_path" + ) + + if re.search("^https", ckpt_path): + ckpt_path = download_file(ckpt_path, self.cache_dirpath) + if re.search("^https", dict_path): + dict_path = download_file(dict_path, self.cache_dirpath) + + model, saved_cfg, _ = fairseq.checkpoint_utils.load_model_ensemble_and_task( + [ckpt_path], + arg_overrides={ + "task": "audio_finetuning", + "data": self.cache_dirpath.as_posix(), + }, # data must have dict in it + ) + + dict_lines = open(dict_path, "r").readlines() + tokens = [l.split()[0] for l in dict_lines] + # adding default fairseq special tokens + tokens = ["", "", "", ""] + tokens + + self.model = model[0] + self.tokens = tokens + + if "|" in tokens: + self.sil_token = "|" + else: + self.sil_token = tokens[ + 2 + ] # use eos as silence token if | not presented e.g., Hok ASR model + print(f"Inferring silence token from the dict: {self.sil_token}") + self.blank_token = self.tokens[0] + + self.sampling_rate = saved_cfg.task.sample_rate + self.normalize_input = saved_cfg.task.normalize + + @torch.inference_mode() + def load_audiofile(self, audio_path: str) -> torch.Tensor: + """ + Load the audio files and apply resampling and normalizaion + + Args: + audio_path: the audio file path + + Returns: + audio_waveform: the audio waveform as a torch.Tensor object + """ + + audio_waveform, sampling_rate = torchaudio.load(audio_path) + if audio_waveform.dim == 2: + audio_waveform = audio_waveform.mean(-1) + if self.sampling_rate != sampling_rate: + audio_waveform = torchaudio.functional.resample( + audio_waveform, sampling_rate, self.sampling_rate + ) + if self.normalize_input: + # following fairseq raw audio dataset + audio_waveform = torch.nn.functional.layer_norm( + audio_waveform, audio_waveform.shape + ) + + return audio_waveform + + @torch.inference_mode() + def compute_emissions(self, audio_input: torch.Tensor) -> torch.Tensor: + """ + Compute the emissions for either fairseq or huggingface asr model + + Args: + audio_path: the input audio waveform + + Returns: + emissions: the logits of the encoded prediction. + """ + + if self.use_cuda: + audio_input = audio_input.to("cuda") + if isinstance(self.model, fairseq.models.wav2vec.wav2vec2_asr.Wav2VecCtc): + padding_mask = lengths_to_padding_mask(torch.tensor([audio_input.numel()])) + emissions = self.model.w2v_encoder(audio_input, padding_mask)[ + "encoder_out" + ].transpose(0, 1) + else: + emissions = self.model(audio_input).logits + + return emissions + + def decode_emissions(self, emissions: torch.Tensor) -> str: + """ + Decode the emissions and apply post process functions + + Args: + emissions: the input Tensor object + + Returns: + hypo: the str as the decoded transcriptions + """ + + emissions = emissions.cpu() + results = self.decoder(emissions) + + # assuming the lexicon-free decoder and working with tokens + hypo = self.decoder.idxs_to_tokens(results[0][0].tokens) + hypo = self.post_process_fn(hypo) + + return hypo + + def transcribe_audiofile(self, audio_path: str, lower=True) -> str: + """ + Transcribe the audio into string + + Args: + audio_path: the input audio waveform + lower: the case of the transcriptions with lowercase as the default + + Returns: + hypo: the transcription result + """ + + asr_input = self.load_audiofile(audio_path) + emissions = self.compute_emissions(asr_input) + hypo = self.decode_emissions(emissions) + + return hypo.strip().lower() if lower else hypo.strip() diff --git a/data/fairseq/examples/speech_to_speech/benchmarking/README.md b/data/fairseq/examples/speech_to_speech/benchmarking/README.md new file mode 100644 index 0000000000000000000000000000000000000000..c62fe12963a20a09078d3c79c65f3ebedf1aa3f9 --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/benchmarking/README.md @@ -0,0 +1,31 @@ +# Benchmarking + +## Overview + +The goal of this framework is to support benchmarking various speech to speech translation(S2ST) models in terms of runtime, max-memory consumption and total number of floating point operations(FLOPS). It is a generic framework and can be easily extended to support any fairseq models. To accurately benchmark the performance, core inference modules are re-implemented based on fairseq_cli/generate.py (core.py/Processing) and examples/speech_to_text/generate_waveform.py(core.py/SpeechGeneration. To ensure that the end to end models and cascaded models are compared fairly, for cascaded models we only consider the performance metrics for model inference at all stages ignoring any intermediate data and io processing consumption. We run all the benchmarking runs on CPU as it is generally used in production environment and also due to lack of good benchmarking library support for GPUs. + +1. Runtime: Average time in seconds to run model inference on an example from a given dataset. We use [timeit](https://docs.python.org/3/library/timeit.html) library to measure the runtime. +2. Max memory: Maximum memory in MiB averaged over by running the model inference on all examples from the given dataset. We use [memory_profiler](https://pypi.org/project/memory-profiler/) library to gather memory footprints for a code snippet and find the maximum to get the max memory used by the code. For cascaded models, we find the max of all stages to get the overall max_memory footprint. +3. FLOPS: We compute the average number of floating point operations needed to run model inference for an example from the given dataset. We use [PAPI library](http://www.bnikolic.co.uk/blog/python/flops/2019/10/01/pytorch-count-flops.html) to benchmark the number of flops. + +## CLI Commands + +```{python} +CUBLAS_WORKSPACE_CONFIG=:4096:8 python examples/speech_to_speech/benchmarking/get_metrics.py ‘’ --config $config +``` + + +## Note: + +1. The npy dataset is a list of samples saved as a .npy file. Each sample is a dictionary with id, net_input. +2. The raw dataset is a list of raw audio paths similar to wav2vec2 input tsv file + +```{python} +sample: { + "id": xx, + "net_input": { + "src_tokens": torch.tensor([]), + "src_lengths": torch.tensor([]) + } +} +``` diff --git a/data/fairseq/examples/speech_to_speech/benchmarking/configs/2StageS2ST.yaml b/data/fairseq/examples/speech_to_speech/benchmarking/configs/2StageS2ST.yaml new file mode 100644 index 0000000000000000000000000000000000000000..11deb42e7d63cf4783d19a6fe17fb8b8cd483aff --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/benchmarking/configs/2StageS2ST.yaml @@ -0,0 +1,19 @@ +general: + dataset_path: $npy_dataset + cpu: True + model_type: 2StageS2ST + dataset_size: 1 + +stage1: + data: $data_bin_stage1 + task: speech_to_text + path: $checkpoint_stage1 + config_yaml: config.yaml + max_len_a: 2 + max_len_b: 500 + +stage2: + data: $data_bin_stage2 + task: text_to_speech + path: $checkpoint_stage2 + config_yaml: config.yaml diff --git a/data/fairseq/examples/speech_to_speech/benchmarking/configs/3StageS2ST.yaml b/data/fairseq/examples/speech_to_speech/benchmarking/configs/3StageS2ST.yaml new file mode 100644 index 0000000000000000000000000000000000000000..96381361508ce5d459f9020ccaf2b18a2fc18aaa --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/benchmarking/configs/3StageS2ST.yaml @@ -0,0 +1,28 @@ +general: + dataset_path: $npy_dataset + cpu: True + model_type: 3StageS2ST + max_len_a: 2 + max_len_b: 500 + dataset_size: 1 + +stage1: + data: $data_bin_stage1 + task: speech_to_text + path: $checkpoint_stage1 + config_yaml: config.yaml + max_len_a: 2 + max_len_b: 500 + +stage2: + data: $data_bin_stage2 + task: translation + path: $checkpoint_stage2 + config_yaml: config.yaml + + +stage2: + data: $data_bin_stage3 + task: text_to_speech + path: $checkpoint_stage3 + config_yaml: config.yaml diff --git a/data/fairseq/examples/speech_to_speech/benchmarking/configs/DirectS2U.yaml b/data/fairseq/examples/speech_to_speech/benchmarking/configs/DirectS2U.yaml new file mode 100644 index 0000000000000000000000000000000000000000..96264cec685ba9d0371318866fcc6c82fffdc02d --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/benchmarking/configs/DirectS2U.yaml @@ -0,0 +1,22 @@ +general: + dataset_path: $npy_dataset_path + cpu: True + model_type: S2UT + dataset_size: 5 + dump_speech_waveforms_dir: $dump_waveforms_dir_path + +stage1: + data: $data_bin + task: speech_to_speech + path: $checkpoint + config_yaml: config.yaml + max_len_b: 100000 + beam: 10 + target_is_code: True + max_target_positions: 3000 + target_code_size: 100 + +stage2: + vocoder: $vocoder_path + vocoder_cfg: $vocoder_cfg_json + dur_prediction: True diff --git a/data/fairseq/examples/speech_to_speech/benchmarking/configs/S2T.yaml b/data/fairseq/examples/speech_to_speech/benchmarking/configs/S2T.yaml new file mode 100644 index 0000000000000000000000000000000000000000..3a106a0441007331e926870a7310e114daf598a0 --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/benchmarking/configs/S2T.yaml @@ -0,0 +1,13 @@ +general: + dataset_path: $npy_dataset + cpu: True + model_type: S2T + dataset_size: 1 + +stage1: + data: $data_bin + task: speech_to_text + path: $checkpoint + config_yaml: config.yaml + max_len_a: 2 + max_len_b: 500 diff --git a/data/fairseq/examples/speech_to_speech/benchmarking/core.py b/data/fairseq/examples/speech_to_speech/benchmarking/core.py new file mode 100644 index 0000000000000000000000000000000000000000..da22a34eced5a1a286855b5856c0c1444b58e5ea --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/benchmarking/core.py @@ -0,0 +1,487 @@ +import timeit +import logging +import torch +from pypapi import events, papi_high as high +from memory_profiler import memory_usage +from torch import nn +from argparse import Namespace +from fairseq.dataclass.utils import convert_namespace_to_omegaconf +from fairseq.data import data_utils as fairseq_data_utils +from fairseq import checkpoint_utils, tasks, utils +from fairseq.models.text_to_speech.vocoder import CodeHiFiGANVocoder +from examples.hubert.simple_kmeans.dump_hubert_feature import HubertFeatureReader +from examples.hubert.simple_kmeans.dump_km_label import ApplyKmeans +from fairseq_cli.generate import get_symbols_to_strip_from_output +import soundfile as sf +import ast +import json + +logging.basicConfig() +logging.root.setLevel(logging.INFO) +logging.basicConfig(level=logging.INFO) +logger = logging.getLogger(__name__) + + +torch.manual_seed(1) +torch.set_deterministic(True) + + +class BenchmarkingBase(nn.Module): + def __init__(self): + nn.Module.__init__(self) + self.s2x_task = None + + def warm_up(self, sample, repeat): + """Warm up the model""" + for _i in range(repeat): + self.forward(sample) + logger.info(f"Model warmed up by running inference {repeat} times") + + def benchmark_run_time(self, dataset, repeat): + """Benchmark average runtime for the model by calling benchmark_run_time_single_sample function""" + logger.info("Starting run time benchmarking") + time_elapsed = 0 + for i, sample in enumerate(dataset): + time_elapsed += self.benchmark_run_time_single_sample(sample, repeat=repeat) + if i % 100 == 0: + logger.info(f"Benchmarked run time for {i}/{len(dataset)} samples") + total_time_elapsed = time_elapsed / len(dataset) + return total_time_elapsed + + def benchmark_run_time_single_sample(self, sample, repeat): + """Benchmark average runtime for a single sample using timeit library. Units are seconds""" + timer = timeit.Timer(lambda: self.forward(sample)) + time_elapsed = timer.timeit(repeat) + return time_elapsed / repeat + + def count_flops( + self, + dataset, + repeat, + ): + """Use PYPAPI library to count average flops for model inference. + Note: It only works if the model is being run on cpu""" + logger.info("Starting flop counter") + high.start_counters([events.PAPI_DP_OPS]) + for i, sample in enumerate(dataset): + for _r in range(repeat): + self.forward(sample) + if i % 100 == 0: + logger.info(f"Counted flops for {i}/{len(dataset)} samples") + flops = high.stop_counters() + flops = round(flops[0] / (repeat * len(dataset))) + return flops + + def max_memory(self, dataset, repeat): + """Compute average max memory consumed by model inference. Units are MiB""" + logger.info("Starting memory benchmarking") + total_memory = 0 + for i, sample in enumerate(dataset): + for _r in range(repeat): + total_memory += max(memory_usage((self.forward, (sample,), {}))) + if i % 100 == 0: + logger.info(f"Benchmarked memory for {i}/{len(dataset)} samples") + total_memory = total_memory / (repeat * len(dataset)) + return total_memory + + def gather_all_metrics(self, dataset, repeat): + run_time = self.benchmark_run_time(dataset, repeat) + max_memory = self.max_memory(dataset, repeat) + flops = self.count_flops(dataset, repeat) + + return run_time, max_memory, flops + + def dump_final_speech_output( + self, dataset, output_dir, resample_fn, sample_rate, prefix=None + ): + + for i, sample in enumerate(dataset): + hypo = self.forward(sample)[0] + + def to_np(x): + return x.detach().cpu().numpy() + + try: + wave_preds = to_np(resample_fn(hypo["waveform"])) + sf.write( + f"{output_dir}/{prefix}_{i}_pred.wav", + wave_preds, + sample_rate, + ) + except Exception as e: + raise Exception( + f" Encountered {e} - Invalid waveform. Make sure the model outputs a waveform" + ) + + +class Processing(BenchmarkingBase): + """Class similar to fairseq_cli/generate.py. Supports ASR, MT and ST model inference""" + + def __init__(self, args): + super().__init__() + self.use_cuda = not getattr(args, "cpu", False) + self.setUp(args) + self.training = False + self.s2x_task = self.task + + def setUp(self, cfg): + if isinstance(cfg, Namespace): + cfg = convert_namespace_to_omegaconf(cfg) + + self.task = tasks.setup_task(cfg.task) + self.tgt_dict = self.task.target_dictionary + + # Load ensemble + logger.info("loading model(s) from {}".format(cfg.common_eval.path)) + models, _ = checkpoint_utils.load_model_ensemble( + utils.split_paths(cfg.common_eval.path), + arg_overrides={}, + task=self.task, + suffix=cfg.checkpoint.checkpoint_suffix, + strict=False, + num_shards=cfg.checkpoint.checkpoint_shard_count, + ) + if len(models) > 1: + raise Exception("Currently loading multiple models is not supported") + self.model = models[0] + + # Optimize model for generation + if cfg.common.fp16: + self.model.half() + if self.use_cuda: + self.model.cuda() + self.model.prepare_for_inference_(cfg) + + self.generator = self.task.build_generator( + [self.model], + cfg.generation, + extra_gen_cls_kwargs={}, + ) + # Handle tokenization and BPE + self.tokenizer = self.task.build_tokenizer(cfg.tokenizer) + self.bpe = self.task.build_bpe(cfg.bpe) + self.remove_bpe = cfg.common_eval.post_process + + def encode_source(self, src): + """Method to generate source tokens from a string""" + if self.tokenizer is not None: + src = self.tokenizer.encode(src) + if self.bpe is not None: + src = self.bpe.encode(src) + src_tokens = self.task.source_dictionary.encode_line(src).long() + src_lens = src_tokens.size(0) + return { + "net_input": { + "src_tokens": src_tokens.view(1, src_lens), + "src_lengths": torch.tensor([src_lens]), + } + } + + def decode_target(self, hypos): + """Method to decode target string from tokens""" + hypo_str = self.tgt_dict.string( + hypos[0][0]["tokens"].int().cpu(), + self.remove_bpe, + get_symbols_to_strip_from_output(self.generator), + ) + if self.bpe is not None: + hypo_str = self.bpe.decode(hypo_str) + if self.tokenizer is not None: + hypo_str = self.tokenizer.decode(hypo_str) + return hypo_str + + def forward(self, sample): + hypos = self.task.inference_step( + self.generator, + [self.model], + sample, + prefix_tokens=None, + constraints=None, + ) + return hypos + + +class GenerateWaveformFromCode(BenchmarkingBase): + """Class to support waveform generation from code. Currently, vocoder only supports single speaker""" + + def __init__(self, args): + super().__init__() + with open(args.vocoder_cfg) as f: + vocoder_cfg = json.load(f) + self.dur_prediction = args.dur_prediction + self.vocoder = CodeHiFiGANVocoder(args.vocoder, vocoder_cfg) + + def format_units(self, input): + code = torch.LongTensor(list(map(int, input.strip().split()))).view(1, -1) + return {"code": code} + + def generate_vocoder_input(self, dataset): + return [self.format_units(sample) for sample in dataset] + + def forward(self, sample): + return [{"waveform": self.vocoder(sample, self.dur_prediction)}] + + +class HubertUnitExtractor(BenchmarkingBase): + def __init__(self, args): + self.feature_reader = HubertFeatureReader( + args.hubert_ckpt_path, args.hubert_layer + ) + self.kmeans = ApplyKmeans(args.hubert_km_path) + + def forward(self, sample): + with torch.no_grad(): + feat = [] + for start in range(0, sample.size(1), self.feature_reader.max_chunk): + x_chunk = sample[:, start : start + self.max_chunk] + feat_chunk, _ = self.feature_reader.model.extract_features( + source=x_chunk, + padding_mask=None, + mask=False, + output_layer=self.layer, + ) + feat.append(feat_chunk) + torch.cat(feat, 1).squeeze(0) + return self.kmeans(feat).tolist() + + +class SpeechGeneration(BenchmarkingBase): + """Class similar to examples/text_to_speech/generate_waveform.py. + Supports models with speech generation as end goal (TTS, Direct S2ST models etc)""" + + def __init__(self, args): + super().__init__() + self.use_cuda = not getattr(args, "cpu", False) + self.setUp(args) + self.s2x_task = self.task + + def setUp(self, args): + if args.task == "speech_to_speech": + args.normalize_waveform = False + self.task = tasks.setup_task(args) + self.pre_tokenizer = self.task.build_tokenizer(args) + self.bpe_tokenizer = self.task.build_bpe(args) + try: + self.src_dict = self.task.src_dict + except Exception: + self.src_dict = None + ensemble, saved_cfg, task = checkpoint_utils.load_model_ensemble_and_task( + [args.path], + arg_overrides=ast.literal_eval(args.model_overrides), + task=self.task, + strict=False, + ) + self.model = ensemble[0] + if self.use_cuda: + self.model.cuda() + # criterion.cuda() + self.model.eval() + self.generator = self.task.build_generator( + [self.model], + args, + ) + + def processTextInput(self, text): + """Generate source tokens from text input""" + if self.pre_tokenizer is not None: + text = self.pre_tokenizer.encode(text) + if self.bpe_tokenizer is not None: + text = self.bpe_tokenizer.encode(text) + target = self.src_dict.encode_line( + text, add_if_not_exist=False, append_eos=True + ).long() + target = fairseq_data_utils.collate_tokens( + [target], + self.src_dict.pad(), + self.src_dict.eos(), + left_pad=False, + move_eos_to_beginning=False, + ) + src_lengths = torch.tensor([target.size(1)], dtype=torch.long) + prev_output_tokens = None + sample = { + "net_input": { + "src_tokens": target, + "src_lengths": src_lengths, + "prev_output_tokens": prev_output_tokens, + } + } + sample = utils.move_to_cuda(sample) if self.use_cuda else sample + return sample + + def forward(self, sample): + sample["speaker"] = None + output = self.generator.generate(self.model, sample) # , has_targ=False + return output + + +class S2UT(BenchmarkingBase): + """Class to support S2UT models. Also supports generating waveforms from the units predicted""" + + def __init__(self, s2u_args, vocoder_args=None): + super().__init__() + self.s2u = Processing(s2u_args) + self.vocoder = None + if vocoder_args: + self.vocoder = GenerateWaveformFromCode(vocoder_args) + self.vocoder_input = None + + def forward(self, sample): + s2u_hypos = self.s2u(sample) + s2u_output = self.s2u.decode_target(s2u_hypos) + if not self.vocoder: + return s2u_output + units = self.vocoder.format_units(s2u_output) + vocoder_output = self.vocoder(units) + return vocoder_output + + def generate_s2u_outputs(self, dataset): + return [self.s2u.decode_target(self.s2u(sample)) for sample in dataset] + + def compute_metrics(self, metric_type, dataset, repeat=None): + """Generic function to compute metrics ignoring the io processing time""" + if self.vocoder and not self.vocoder_input: + self.s2u_output = self.generate_s2u_outputs(dataset) + self.vocoder_input = self.vocoder.generate_vocoder_input(self.s2u_output) + + s2u_metrics = getattr(self.s2u, metric_type)( + dataset, + repeat, + ) + vocoder_metrics = 0 + if self.vocoder: + vocoder_metrics = getattr(self.vocoder, metric_type)( + self.vocoder_input, + repeat, + ) + print( + f"metric_type = {metric_type} s2u_metrics = {s2u_metrics} \t vocoder_metrics = {vocoder_metrics}" + ) + if metric_type == "max_memory": + return max(s2u_metrics, vocoder_metrics) + else: + return s2u_metrics + vocoder_metrics + + def benchmark_run_time(self, dataset, repeat): + return self.compute_metrics("benchmark_run_time", dataset, repeat) + + def count_flops(self, dataset, repeat): + return self.compute_metrics("count_flops", dataset, repeat) + + def max_memory(self, dataset, repeat): + return self.compute_metrics("max_memory", dataset, repeat) + + +class Cascaded2StageS2ST(BenchmarkingBase): + """ST + TTS""" + + def __init__(self, s2t_args, tts_args): + super().__init__() + self.s2t = Processing(s2t_args) + self.s2x_task = self.s2t.task + self.tts = SpeechGeneration(tts_args) if tts_args else None + self.training = False + self.tts_inputs = None + + def forward(self, sample): + if not self.tts: + raise Exception( + "Forward function is not callable without tts. Reinitialize the class with tts_args" + ) + s2t_hypos = self.s2t(sample) + s2t_output = self.s2t.decode_target(s2t_hypos) + tts_input = self.tts.processTextInput(s2t_output) + tts_output = self.tts(tts_input) + return tts_output + + def generate_s2t_outputs(self, dataset): + """Process dataset and generate s2t outputs""" + return [self.s2t.decode_target(self.s2t(sample)) for sample in dataset] + + def generate_tts_inputs(self, dataset): + """Process dataset and generate tts inputs""" + return [self.tts.processTextInput(sample) for sample in dataset] + + def compute_metrics(self, metric_type, dataset, repeat=None): + """Generic function to compute metrics ignoring the io processing time""" + if not self.tts_inputs: + s2t_outputs = self.generate_s2t_outputs(dataset) + self.tts_inputs = self.generate_tts_inputs(s2t_outputs) + + s2t_metrics = getattr(self.s2t, metric_type)( + dataset, + repeat, + ) + + tts_metrics = getattr(self.tts, metric_type)( + self.tts_inputs, + repeat, + ) + print( + f"metric_type = {metric_type} s2t_metrics = {s2t_metrics} \t tts_metrics = {tts_metrics}" + ) + if metric_type == "max_memory": + return max(s2t_metrics, tts_metrics) + else: + return s2t_metrics + tts_metrics + + def benchmark_run_time(self, dataset, repeat): + return self.compute_metrics("benchmark_run_time", dataset, repeat) + + def count_flops(self, dataset, repeat): + return self.compute_metrics("count_flops", dataset, repeat) + + def max_memory(self, dataset, repeat): + return self.compute_metrics("max_memory", dataset, repeat) + + +class Cascaded3StageS2ST(Cascaded2StageS2ST): + """ASR + MT + TTS""" + + def __init__(self, s2t_args, tts_args, mt_args): + super().__init__(s2t_args, tts_args) + self.mt = Processing(mt_args) + self.mt_inputs = [] + + def forward(self, sample): + s2t_hypos = self.s2t(sample) + s2t_output = self.s2t.decode_target(s2t_hypos) + mt_input = self.mt.encode_source(s2t_output) + mt_hypos = self.mt(mt_input) + mt_output = self.mt.decode_target(mt_hypos) + tts_input = self.tts.processTextInput(mt_output) + tts_output = self.tts(tts_input) + return tts_output + + def generate_mt_inputs(self, dataset): + """Process dataset to generate mt model inputs""" + return [self.mt.encode_source(sample) for sample in dataset] + + def generate_mt_outputs(self, dataset): + """Process dataset to generate mt model outputs""" + return [self.mt.decode_target(self.mt(sample)) for sample in dataset] + + def compute_metrics(self, metric_type, dataset, repeat=None): + """Generic function to compute metrics ignoring the io processing time""" + if not self.tts_inputs: + s2t_outputs = self.generate_s2t_outputs(dataset) + self.mt_inputs = self.generate_mt_inputs(s2t_outputs) + mt_outputs = self.generate_mt_outputs(self.mt_inputs) + self.tts_inputs = self.generate_tts_inputs(mt_outputs) + + s2t_metrics = getattr(self.s2t, metric_type)( + dataset, + repeat, + ) + mt_metrics = getattr(self.mt, metric_type)(self.mt_inputs, repeat) + tts_metrics = getattr(self.tts, metric_type)( + self.tts_inputs, + repeat, + ) + print( + f"metric_type = {metric_type} s2t_metrics = {s2t_metrics} \t mt_metrics = {mt_metrics} \t tts_metrics = {tts_metrics}" + ) + if metric_type == "max_memory": + return max(s2t_metrics, mt_metrics, tts_metrics) + else: + return s2t_metrics + mt_metrics + tts_metrics diff --git a/data/fairseq/examples/speech_to_speech/benchmarking/data_utils.py b/data/fairseq/examples/speech_to_speech/benchmarking/data_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..c73a59951f5862c5aa8dffc901a950b5b36272f7 --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/benchmarking/data_utils.py @@ -0,0 +1,264 @@ +from fairseq import tasks +import numpy as np +import logging +import random +from fairseq import options +import torch +import os +import soundfile as sf + +from fairseq.data.audio.audio_utils import ( + get_waveform, + parse_path, +) + +logging.basicConfig() +logging.root.setLevel(logging.INFO) +logging.basicConfig(level=logging.INFO) +logger = logging.getLogger(__name__) + +random.seed(1) +np.random.seed(1) +random_number_generator = np.random.RandomState(30) + + +def generate_random_data_sample(T, B=1, D=80): + """Generate random data sample given the T, B, D values""" + net_input = { + "src_tokens": torch.tensor(random_number_generator.randn(B, T, D)).float(), + "src_lengths": torch.tensor([T]), + } + return {"net_input": net_input} + + +def generate_random_dataset(T_range_min, T_range_max, B=1, D=80, dataset_size=100): + """Generate random dataset with T values within a given range, B, D""" + T_values = [random.randint(T_range_min, T_range_max) for i in range(dataset_size)] + dataset = [] + for t in T_values: + dataset.append(generate_random_data_sample(t, B, D)) + return dataset, sum(T_values) / dataset_size + + +def load_dataset_npy(file_name, dataset_size=None): + """Load dataset from a .npy file.""" + data = np.load(file_name, allow_pickle=True) + if dataset_size: + data = data[:dataset_size] + return data + + +def load_dataset_raw_to_waveforms( + file_name, + dataset_size=None, + need_waveform=True, + sample_rate=16000, + read_using_soundfile=False, +): + """Load raw dataset from w2v tsv file. Optionally get waveforms""" + data = [] + with open(file_name, "r") as fp: + lines = fp.readlines() + data = [ + os.path.join(lines[0].strip(), line.strip().split("\t")[0]) + for line in lines[1:] + ] + + if dataset_size: + data = data[:dataset_size] + + if not need_waveform: + return data + + features = [] + if read_using_soundfile: + for _i, d in enumerate(data): + wav = sf.read(d)[0] + if wav.ndim == 2: + wav = wav.mean(-1) + features.append(torch.from_numpy(wav).float().view(1, -1)) + else: + for i, d in enumerate(data): + _path, slice_ptr = parse_path(d) + if len(slice_ptr) == 0: + feat = get_waveform( + _path, always_2d=True, output_sample_rate=sample_rate + )[0] + features.append( + { + "id": i, + "net_input": { + "src_tokens": torch.tensor(feat), + "src_lengths": torch.tensor([feat.shape[1]]), + }, + } + ) + else: + raise Exception("Currently unsupported data format") + return features + + +def load_dataset_task( + args, + batch_size=1, + limit_size=None, + ref_dataset=None, +): + """Loads dataset based on args by creating a task""" + if not args.data or not args.subset or not args.task: + raise Exception( + "Please provide necessary arguments to load the dataset - data, subset and task" + ) + task = tasks.setup_task(args) + + task.load_dataset(args.subset) + if not limit_size: + limit_size = len(task.dataset(args.subset)) + + iter = task.get_batch_iterator( + dataset=task.dataset(args.subset), max_sentences=batch_size + ).next_epoch_itr(shuffle=False) + dataset = [] + for i, sample in enumerate(iter): + sample = { + "id": task.datasets[args.subset].ids[sample["id"].item()], + "net_input": { + "src_tokens": sample["net_input"]["src_tokens"], + "src_lengths": sample["net_input"]["src_lengths"], + }, + } + dataset.append(sample) + if i == limit_size - 1: + break + + if ref_dataset: + try: + ids = get_ids_from_dataset(ref_dataset) + except Exception as e: + raise Exception(f"{e} - Cannot extract ids from reference dataset") + + filtered_dataset = [] + for sample in dataset: + if ( + sample["id"] in ids + or sample["id"][5:] in ids + or f"dev_{sample['id']}" in ids + ): + filtered_dataset.append(sample) + dataset = filtered_dataset + + max_len, min_len, avg_len = get_dataset_stats(dataset) + print( + f"{args.subset} dataset stats : num_samples={len(dataset)} max_len = {max_len} min_len = {min_len} avg_len = {avg_len}" + ) + + return dataset + + +def randomly_sample_subset(dataset, size=500): + """Randomly sample subset from a dataset""" + random_indices = [random.randint(0, len(dataset) - 1) for i in range(size)] + return [dataset[i] for i in random_indices] + + +def get_short_data_subset(dataset, size=500): + """Get a subset of desired size by sorting based on src_lengths""" + return sort_dataset(dataset)[:size] + + +def get_long_data_subset(dataset, size=500): + """Get a subset of desired size by sorting based on src_lengths descending""" + return sort_dataset(dataset, reverse=True)[:size] + + +def sort_dataset(dataset, reverse=False): + return sorted( + dataset, key=lambda x: x["net_input"]["src_lengths"].item(), reverse=reverse + ) + + +def save_dataset_npy(dataset, file_name): + """Save a dataset as .npy file""" + np.save(file_name, dataset) + + +def get_dataset_stats(dataset): + """Get stats about dataset based on src_lengths of samples""" + max_len = 0 + min_len = 100000 + avg_len = 0 + for d in dataset: + max_len = max(max_len, d["net_input"]["src_lengths"].item()) + min_len = min(min_len, d["net_input"]["src_lengths"].item()) + avg_len += d["net_input"]["src_lengths"].item() + + return max_len, min_len, avg_len / len(dataset) + + +def make_parser(): + """ + Additional args: + 1. Provide the dataset dir path using --data. + 2. Loading the dataset doesn't require config, provide --config-yaml to apply additional feature transforms + """ + parser = options.get_speech_generation_parser() + parser.add_argument( + "--subset", + default=None, + type=str, + required=True, + help="Subset to use for dataset generation", + ) + parser.add_argument( + "--dataset-save-dir", + default=None, + type=str, + required=False, + help="Dir path in which the datasets are to be saved", + ) + parser.add_argument( + "--ref-dataset", + default=None, + type=str, + required=False, + help="If provided, the ids in the reference dataset will be used to filter the new dataset generated.", + ) + parser.add_argument("--dataset-save-token", default="", type=str, required=False) + + options.add_generation_args(parser) + return parser + + +def get_ids_from_dataset(dataset): + return {sample["id"]: 1 for sample in dataset} + + +def cli_main(): + parser = make_parser() + args = options.parse_args_and_arch(parser) + dataset = load_dataset_task(args) + + random_dataset = randomly_sample_subset(dataset) + short_dataset = get_short_data_subset(dataset) + long_dataset = get_long_data_subset(dataset) + + if args.dataset_save_token: + args.dataset_save_token = f"_{args.dataset_save_token}_" + + if args.dataset_save_dir: + save_dataset_npy( + random_dataset, + f"{args.dataset_save_dir}/random_dataset{args.dataset_save_token}w_ids.npy", + ) + save_dataset_npy( + short_dataset, + f"{args.dataset_save_dir}/short_dataset{args.dataset_save_token}w_ids.npy", + ) + save_dataset_npy( + long_dataset, + f"{args.dataset_save_dir}/long_dataset{args.dataset_save_token}w_ids.npy", + ) + + +if __name__ == "__main__": + cli_main() diff --git a/data/fairseq/examples/speech_to_speech/benchmarking/get_metrics.py b/data/fairseq/examples/speech_to_speech/benchmarking/get_metrics.py new file mode 100644 index 0000000000000000000000000000000000000000..773257f5daa970ac73281c4328fdf977022a02f2 --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/benchmarking/get_metrics.py @@ -0,0 +1,162 @@ +import copy +import torch +import logging +from argparse import Namespace +import yaml +from fairseq import options +from examples.speech_to_speech.benchmarking.core import ( + Processing, + SpeechGeneration, + Cascaded2StageS2ST, + Cascaded3StageS2ST, + S2UT, +) +from examples.speech_to_speech.benchmarking.data_utils import ( + load_dataset_npy, + load_dataset_raw_to_waveforms, +) + + +logging.basicConfig() +logging.root.setLevel(logging.INFO) +logging.basicConfig(level=logging.INFO) +logger = logging.getLogger(__name__) + +torch.manual_seed(1) +torch.set_deterministic(True) + + +def make_parser(): + """Note: As the names indicate use s2x_args(ex:ST, ASR etc) for models with speech input, + x2s_args for models with speech output(ex:TTS) and mt_args for translation models (ex: mt, T2U etc). + For direct S2ST models, use x2s_args to provide model details. + """ + parser = options.get_speech_generation_parser() + parser.add_argument("--target-is-code", action="store_true", default=False) + parser.add_argument("--config", type=str) + parser.add_argument( + "--model-type", + default="S2U", + choices=["S2S", "TTS", "S2UT", "MT", "S2T", "2StageS2ST", "3StageS2ST"], + help="Choose one of the models. For model inference implementation, refer to core.py", + ) + parser.add_argument( + "--dataset-path", + type=str, + help="""File to load dataset from. Assumes dataset is a list of samples. + Each sample is a dict of format {'net_input':{'src_tokens':torch.tenor(),'src_lengths':torch.tensor()}}""", + ) + parser.add_argument( + "--dataset-type", + type=str, + default="npy", + choices=["npy", "raw"], + help="""Type of input dataset file""", + ) + parser.add_argument( + "--read-using-sf", + type=str, + default=False, + help="""If sound file should be used to read the raw dataset""", + ) + parser.add_argument( + "--dataset-size", + default=None, + type=int, + help="Dataset size to use for benchmarking", + ) + parser.add_argument( + "--dump-speech-waveforms-dir", + default=None, + type=str, + help="Directory to dump the speech waveforms computed on the dataset.", + ) + parser.add_argument( + "--dump-waveform-file-prefix", + default="", + type=str, + help="File name prefix for the saved speech waveforms", + ) + parser.add_argument( + "--feat-dim", default=80, type=int, help="Input feature dimension" + ) + parser.add_argument( + "--target-sr", + default=16000, + type=int, + help="Target sample rate for dumping waveforms", + ) + + options.add_generation_args(parser) + options.get_interactive_generation_parser(parser) + return parser + + +def cli_main(): + parser = make_parser() + args = options.parse_args_and_arch(parser) + + with open( + args.config, + "r", + ) as f: + config = yaml.load(f, Loader=yaml.FullLoader) + dict_args = vars(args) + dict_args.update(config["general"]) + args = Namespace(**dict_args) + + i = 1 + stage_args = [] + while i <= 3: + var = f"stage{i}" + tmp_args = copy.deepcopy(dict_args) + if var in config: + tmp_args.update(config[var]) + stage_args.append(Namespace(**tmp_args)) + i += 1 + else: + break + + if args.model_type == "S2S" or args.model_type == "TTS": + model = SpeechGeneration(stage_args[0]) + elif args.model_type == "S2UT": + model = S2UT(stage_args[0], stage_args[1] if len(stage_args) > 1 else None) + elif args.model_type == "MT" or args.model_type == "S2T": + model = Processing(stage_args[0]) + elif args.model_type == "2StageS2ST": + model = Cascaded2StageS2ST(stage_args[0], stage_args[1]) + elif args.model_type == "3StageS2ST": + model = Cascaded3StageS2ST(stage_args[0], stage_args[2], stage_args[1]) + else: + raise Exception(f"Currently unsupported model type {args.model_type}") + + print(f"Evaluating on dataset - {args.dataset_path}\n") + + if args.dataset_type == "npy": + dataset = load_dataset_npy(args.dataset_path, dataset_size=args.dataset_size) + elif args.dataset_type == "raw": + dataset = load_dataset_raw_to_waveforms( + args.dataset_path, + dataset_size=args.dataset_size, + read_using_soundfile=args.read_using_sf, + ) + else: + raise Exception(f"Invalid dataset type {args.dataset_type}") + + model.warm_up(sample=dataset[0], repeat=2) + + run_time, memory, flops = model.gather_all_metrics(dataset, repeat=1) + print(f"run_time = {run_time}sec \tmemory = {memory}MiB \tflops = {flops}") + + if args.dump_speech_waveforms_dir: + model.dump_final_speech_output( + dataset, + args.dump_speech_waveforms_dir, + lambda x: x, + args.target_sr, + prefix=args.dump_waveform_file_prefix, + ) + + +if __name__ == "__main__": + cli_main() diff --git a/data/fairseq/examples/speech_to_speech/docs/data_augmentation.md b/data/fairseq/examples/speech_to_speech/docs/data_augmentation.md new file mode 100644 index 0000000000000000000000000000000000000000..c0c17ff2235dbfa4c90ad033a53bb7fb5080e3bd --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/docs/data_augmentation.md @@ -0,0 +1,435 @@ +# Noise and audio augmentation techniques + +The noise and data augmentation techniques were written in an effort to understand how augmenatation can affect model robustness and performance in both clean and noisy settings. + +All transforms discussed in this section are subclasses of `AudioFeatureTransform`, `AudioWaveformTransform`, or `AudioDatasetTransform`. Each `Audio*Transform` has unique interaction with the data. If interested in implemented one's own transforms, it is highly advisable to review the differences (see [Adding your own transforms](https://github.com/facebookresearch/fairseq/blob/main/examples/speech_to_speech/docs/data_augmentation.md#adding-your-own-transforms)). If only applying the in-built transforms, then one only needs to be mindful that the correct kind of transform is listed in the config (see [Using transforms](https://github.com/facebookresearch/fairseq/blob/main/examples/speech_to_speech/docs/data_augmentation.md#using-transforms)). These transforms can be applied to instances of `SpeechToTextDataset`. + +### Contents +[In-built transforms](https://github.com/facebookresearch/fairseq/blob/main/examples/speech_to_speech/docs/data_augmentation.md#in-built-transforms) + +[Benchmark studies](https://github.com/facebookresearch/fairseq/blob/main/examples/speech_to_speech/docs/data_augmentation.md#benchmark-studies) + +[Using transforms](https://github.com/facebookresearch/fairseq/blob/main/examples/speech_to_speech/docs/data_augmentation.md#using-transforms) + +[Adding your own transforms](https://github.com/facebookresearch/fairseq/blob/main/examples/speech_to_speech/docs/data_augmentation.md#adding-your-own-transforms) + + +## In-built transforms +### 1. Utterance concatenation +Utterance concatenation is a data augmenation technique introduced as ConcatAug in [Translatotron 2: High-quality direct speech-to-speech translation +with voice preservation](https://arxiv.org/pdf/2107.08661.pdf). +With some parameterized probability, samples are concatenated with one other randomly chosen sample from the whole dataset. In the positive (concatenation) case, accessing `dataset[i]` will return a `SpeechToTextDatasetItem` where `source=source[i]+source[j]` and `target=target[i]+target[j]`. In the negative (skip concatenation) case, accessing `dataset[i]` will return a `SpeechToTextDatasetItem` where `source=source[i]` and `target=target[i]` as usual. + +**Usage**: `concataugment` is an `AudioDatasetTransform` and has three configurable hyperparameters: +- `rate`: probability that any single access will result in the positive (concatenation) case. Defaults to 0.25. +- `max_tokens`: maximum number of tokens allowed for concatenated source sequences. This parameter is meant to limit the length of concatenated samples to avoid out-of-memory errors. Defaults to 300. +- `attempts`: maximum number of invalid concatenation attempts before defaulting to the negative (skip concatenation) case. This parameter aims to limit excessive time spent trying to find candidate samples that are short enough to concatenate with. Defaults to 5. + +Please be wary of OOMs while using this augmentation technique; we used smaller batch sizes as a workaround to avoid OOMs. Batch size is determined by update frequency, batch size hyperparameter, and the number of GPU, so you may want to alter these to this end. + +### 2. Noise augmentation suite + +The four noise augmentation methods in this suite adhere to the following principle: with some parameterized probability, samples are overlayed with a noise track. The content of the noise track is specific to the method. Signal-to-noise ratio with which the noise track is overlayed is determined by choosing a value from a random uniform distribution with parameterized endpoints. The first three methods are based off data augmentation methods suggested in Section 3.3 of [X-Vectors: Robust DNN Embeddings for Speaker Recognition](https://danielpovey.com/files/2018_icassp_xvectors.pdf). + +#### 2.1. Music augmentation +For music augmentation, the noise track consists of one file uniformly randomly selected from a corpus of music files. The music file is cut to size, including being repeated to fill the original sample length if necessary. + +**Usage**: `musicaugment` is an `AudioWaveformTransform` and has four configurable hyperparameters: +- `samples_path`: path where background music files are saved as audios (.wav files). No default. +- `rate`: probability that any single access will result in the positive (background music) case. Defaults to 0.25. +- `snr_min`: lower endpoint of the range from which a signal-to-noise ratio is uniformly randomly chosen with which to add background noise to the original source. Defaults to 5. +- `snr_max`: higher endpoint of the range from which a signal-to-noise ratio is uniformly randomly chosen with which to add background noise to the original source. Defaults to 15. + +#### 2.2. Babble augmentation +For babble augmentation, the noise track consists of multiple audios uniformly randomly selected from a corpus of speech files. The number of speech audios in the background track is chosen randomly with equal probability between 3 and 7 audios. + +**Usage**: `babbleaugment` is an `AudioWaveformTransform` and has four configurable hyperparameters: +- `samples_path`: path where background speech files are saved as audios (.wav files). No default. +- `rate`: probability that any single access will result in the positive (background speech) case. Defaults to 0.25. +- `snr_min`: lower endpoint of the range from which a signal-to-noise ratio is uniformly randomly chosen with which to add background noise to the original source. Defaults to 5. +- `snr_max`: higher endpoint of the range from which a signal-to-noise ratio is uniformly randomly chosen with which to add background noise to the original source. Defaults to 15. + +#### 2.3. Sporadic noise augmentation +For sporadic noise augmentation, the noise track is mostly silent except for intermittent short clips of noise which are added at roughly a parameterized frequency. These clips are randomly chosen and cut from a corpus of noise files to lengths according to a parameterized Gaussian distribution. + +**Usage**: `sporadicnoiseaugment` is an `AudioWaveformTransform` and has seven configurable hyperparameters: +- `samples_path`: path where background noise files are saved as audios (.wav files). No default. +- `rate`: probability that any single access will result in the positive (add a sporadic noise track) case. Defaults to 0.25. +- `snr_min`: lower endpoint of the range from which a signal-to-noise ratio is uniformly randomly chosen with which to add background noise to the original source. Defaults to 5. +- `snr_max`: higher endpoint of the range from which a signal-to-noise ratio is uniformly randomly chosen with which to add background noise to the original source. Defaults to 15. +- `noise_rate`: rate in noises per second at which noise clip will be added to the original sample +- `noise_len_mean`: mean of Gaussian normal distribution from which length of noise clip is chosen +- `noise_len_std`: standard deviation of Gaussian normal distribution from which length of noise clip is chosen + +#### 2.4. Background noise augmentation +For background noise augmentation, the noise track is a single track uniformly randomly selected from a corpus of noise files. The noise file is cut to size, including being repeated to fill the original sample length if necessary. + +**Usage**: `backgroundnoiseaugment` is an `AudioWaveformTransform` and has four configurable hyperparameters: +- `samples_path`: path where background noise files are saved as audios (.wav files). No default. +- `rate`: probability that any single access will result in the positive (background noise) case. Defaults to 0.25. +- `snr_min`: lower endpoint of the range from which a signal-to-noise ratio is uniformly randomly chosen with which to add background noise to the original source. Defaults to 5. +- `snr_max`: higher endpoint of the range from which a signal-to-noise ratio is uniformly randomly chosen with which to add background noise to the original source. Defaults to 15. + +### 3. Mixed babble and background noise augmentation with recognizable source speaker + +This augmentation technique is based on Algorithm 1 in [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://arxiv.org/abs/2110.13900) and is similar to the noise augmentation suite techniques in that it has a background noise track. The noise track consists of either (1) another audio sample from the batch or (2) a background noise track. A key difference is the length of the noise track is chosen from a uniform random distribution between 0 and half of the original sample length. + +**Usage**: `noisyoverlapaugment` is an `AudioDatasetTransform` and has seven configurable hyperparameters: +- `noises_path`: path where background noise files are saved as audios (.wav files). No default. +- `rate`: probability that any single access will result in the positive (background noise) case. Defaults to 0.25. +- `mixing_noise_rate`: probability that in a positive (background noise) case, the noise track will consist of background noise (rather than babble from the batch). Defaults to 0.1. +- `noise_snr_min`: lower endpoint of the range from which a signal-to-noise ratio is uniformly randomly chosen with which to add background noise to the original source. Defaults to -5. +- `noise_snr_max`: higher endpoint of the range from which a signal-to-noise ratio is uniformly randomly chosen with which to add background noise to the original source. Defaults to 5. +- `utterance_snr_min`: lower endpoint of the range from which a signal-to-noise ratio is uniformly randomly chosen with which to add **another audio from the batch** to the original source. Defaults to -5. +- `utterance_snr_max`: higher endpoint of the range from which a signal-to-noise ratio is uniformly randomly chosen with which to add **another audio from the batch** to the original source. Defaults to 5. + +## Benchmark studies +### Evaluation on clean data +Augmentation in training data|Hyperparameters|Training loss|BLEU (covost)|BLEU (epst)|BLEU (mtedx) +---|---|---|---|---|--- +None||3.954|24.984|23.962|24.448 +ConcatAugment|rate = 0.25, max_tokens = 3000, attempts = 5|3.940|25.322|26.124|26.19 +BabbleAugment|rate = 0.25, MUSAN speech, snr_min = (-5), snr_max = 5|3.957|24.226|23.186|22.368| +BackgroundNoiseAugment|rate = 0.1, MUSAN noises, snr_min = (-10), snr_max = 10|3.955|24.745|23.513|23.819 +MusicAugment|rate = 0.25, MUSAN music, snr_min = 0, snr_max = 20|3.954|25.096|24.301|23.341| +SporadicNoiseAugment|rate = 0.1, noise_rate = 0.25, MUSAN noises, snr_min = 10, snr_max = 35|3.954|24.924|23.951|23.484| +MusicAugment + BabbleAugment + BackgroundNoiseAugment + SporadicNoiseAugment|as above, except limited rates to sum to 0.25: music (0.074), background (0.029), babble (0.074), sporadic (0.029)|3.953|24.874|23.675|24.249| +NoisyOverlapAugment|rate = 0.25, mixing_noise_rate = 0.5, MUSAN noises, utterance_snr_min = (-10), utterance_snr_max = 0, noise_snr_min = (-5), noise_snr_max = 20|3.954|24.949|24.015|23.768| + +### Evaluation on data with music noise added at SNR = (-5) - 5 +Augmentation in training data|Training loss|BLEU (covost)|BLEU (epst)|BLEU (mtedx) +---|---|---|---|--- +None|3.954|15.785|21.105|16.944 +ConcatAugment|3.940|17.186|23.255|18.24 +BabbleAugment|3.957|19.158|22.064|17.116 +BackgroundNoiseAugment|3.955|17.777|22.0|17.535| +MusicAugment|3.954|20.345|23.126|19.433| +SporadicNoiseAugment|3.954|15.927|21.382|14.736| +MusicAugment + BabbleAugment + BackgroundNoiseAugment + SporadicNoiseAugment|3.953|19.724|22.659|17.852| +NoisyOverlapAugment|3.954|17.49|22.142|17.207| + +### Evaluation on data with babble noise added at SNR = (-5) - 5 +Augmentation in training data|Training loss|BLEU (covost)|BLEU (epst)|BLEU (mtedx) +---|---|---|---|--- +None|3.954|4.092|13.514|5.13 +ConcatAugment|3.940|5.493|15.835|6.893 +BabbleAugment|3.957|16.12|21.097|13.996 +BackgroundNoiseAugment|3.955|4.691|15.784|5.982 +MusicAugment|3.954|8.06|17.764|9.008 +SporadicNoiseAugment|3.954|4.009|13.935|4.814 +MusicAugment + BabbleAugment + BackgroundNoiseAugment + SporadicNoiseAugment|3.953|14.692|20.882|14.45 +NoisyOverlapAugment|3.954|4.032|16.434|7.284 + +### Evaluation on data with sporadic noise added at SNR = (-5) - 5 +Augmentation in training data|Training loss|BLEU (covost)|BLEU (epst)|BLEU (mtedx) +---|---|---|---|--- +None|3.954|23.778|23.745|22.748 +ConcatAugment|3.940|24.239|25.907|25.723 +BabbleAugment|3.957|23.42|23.048|21.076 +BackgroundNoiseAugment|3.955|23.998|23.467|22.494 +MusicAugment|3.954|24.142|24.181|19.143 +SporadicNoiseAugment|3.954|23.97|23.894|22.61 +MusicAugment + BabbleAugment + BackgroundNoiseAugment + SporadicNoiseAugment|3.953|24.118|23.59|23.717 +NoisyOverlapAugment|3.954|24.265|24.103|23.167 + +### Evaluation on data with background noise added at SNR = (-5) - 5 +Augmentation in training data|Training loss|BLEU (covost)|BLEU (epst)|BLEU (mtedx) +---|---|---|---|--- +None|3.954|20.201|22.525|19.66 +ConcatAugment|3.940|20.904|24.706|21.353 +BabbleAugment|3.957|20.687|22.374|18.907 +BackgroundNoiseAugment|3.955|21.574|22.998|20.043 +MusicAugment|3.954|21.65|23.529|19.87 +SporadicNoiseAugment|3.954|20.578|22.577|19.096 +MusicAugment + BabbleAugment + BackgroundNoiseAugment + SporadicNoiseAugment|3.953|21.811|23.144|20.986 +NoisyOverlapAugment|3.954|21.312|23.153|20.302 + +### Evaluation on data with all four types of noises added at SNR = (-5) - 5, each applied with prob 0.5 +Augmentation in training data|Training loss|BLEU (covost)|BLEU (epst)|BLEU (mtedx) +---|---|---|---|--- +None|3.954|10.895|19.319|12.748 +ConcatAugment|3.940|13.517|21.658|15.428 +BabbleAugment|3.957|18.09|21.384|16.018 +BackgroundNoiseAugment|3.955|12.837|20.719|13.933 +MusicAugment|3.954|16.589|21.823|15.927 +SporadicNoiseAugment|3.954|11.238|19.91|13.31 +MusicAugment + BabbleAugment + BackgroundNoiseAugment + SporadicNoiseAugment|3.953|18.636|21.935|17.845 +NoisyOverlapAugment|3.954|12.829|20.856|15.048 + +### Evaluation on data with noisy overlap augment +Augmentation in training data|Training loss|BLEU (covost)|BLEU (epst)|BLEU (mtedx) +---|---|---|---|--- +None|3.954|21.245|22.24|20.994 +ConcatAugment|3.940|21.611|24.247|23.068 +BabbleAugment|3.957|21.867|21.987|20.099| +BackgroundNoiseAugment|3.955|21.533|21.806|19.717| +MusicAugment|3.954|21.823|22.643|20.847| +SporadicNoiseAugment|3.954|21.373|22.381|20.672| +MusicAugment + BabbleAugment + BackgroundNoiseAugment + SporadicNoiseAugment|3.953|22.206|22.414|21.375| +NoisyOverlapAugment|3.954|23.371|23.396|22.627| + +## Using transforms +Transforms are configurable. + +1. Please pay careful attention to the type of transform you are applying. + - `concataugment` and `noisyoverlapaugment` are instances of `AudioDatasetTransform` and should be listed in the config under `dataset_transforms`. + - `musicaugment`, `babbleaugment`, `sporadicnoiseaugment`, and `backgroundnoiseaugment` are instances of `AudioWaveformTransform` and should be listed under `waveform_transforms`. + - Instances of `AudioFeatureTransform` should be listed under `feature_transforms`. +2. Feel free to apply these augmentations in different contexts, e.g., you may use a `_train` or `_eval` flag to specify when the transform will be applied. If the dataset at hand contains `train` in its name, those transforms under the `_train` flag will be applied; else, the remaining transforms will be applied. + +For example, you would add this to your config to apply the musicaugment transform to a training dataset: +```yaml +musicaugment: + samples_path: ${MUSIC_PATH} + snr_min: 10 + snr_max: 15 + rate: 0.25 +waveform_transforms: + _train: + - musicaugment +``` +or add this to apply the concataugment transform: +```yaml +concataugment: + rate: 0.25 + max_tokens: 3000 + attempts: 5 +dataset_transforms: + _train: + - concataugment + ``` +You may also want to add multiple of one type of transform; here, we add multiple `AudioWaveformTransform`s: +```yaml +musicaugment: + samples_path: ${MUSIC_PATH} + snr_min: 5 + snr_max: 20 + rate: 0.25 +backgroundnoiseaugment: + samples_path: ${NOISES_PATH} + snr_min: 10 + snr_max: 20 + rate: 0.1 +sporadicnoiseaugment: + samples_path: ${NOISES_PATH} + snr_min: 5 + snr_max: 15 + rate: 0.1 + noise_rate: 0.25 +waveform_transforms: + _train: + - musicaugment + - backgroundnoiseaugment + - sporadicnoiseaugment +``` + +## Adding your own transforms +Note: We store transform implementations in `fairseq/data/audio/*_transforms` directories. You may refer to these as examples while implementing your own transform. + +### Step 1. Picking the right class for your transform +The integration into SpeechToTextDataset is quite different for each kind of transform, so it is important to understand which one is best suited to your purposes. + +**Feature transforms** +`AudioFeatureTransform` is a base class which allows **some transform to be applied to audio spectrograms** in the data loading step. One thing to note is that the source data is either saved as `np.ndarrays` or as audio files, and is to be returned either as features (spectrogram) or waveform. If and only if the data is to be returned as a spectrogram, then `AudioFeatureTransform`s will be applied. + +**Waveform transforms** +`AudioWaveformTransform` is a base class which allows some **transform to be applied to waveforms** in the data loading step. As mentioned above, there are two source and return types to data loading for this dataset. If and only if the data is saved in audio file format, then `AudioWaveformTransform`s will be applied, whichever return type is used. + +**Dataset transforms** +`AudioDatasetTransform` is a base class for transforms **based on more than one item in a dataset**, ex. concatenation of two random samples in a dataset. Rather than being applied in a consistent way, i.e., to all features or to all waveforms, the integration of a dataset transform is entirely specific. Adding a dataset transform requires actually editing the `fairseq/data/audio/speech_to_text_dataset.py` file. + +### Step 2. Setting up your transform (generic to all types of transforms) +Now that you know which kind of transform you would like to use, we are ready to implement it. This step is generic for all transform types, i.e., `TRANSFORM_TYPE` may be any of `feature`, `waveform`, or `dataset`. We will show how to build utterance concatenation (an `AudioDatasetTransform`) as an example. + +Import the base class and registration function for your transform. +```python +from fairseq.data.audio.dataset_transforms import ( + AudioDatasetTransform, + register_audio_dataset_transform +) +``` + +Define the class and register the transform. The name passed into the registration function is how your transform should be named in the config. +```python +@register_audio_dataset_transform("concataugment") +class ConcatAugment(AudioDatasetTransform): +``` + +We are now ready to add the basic important functions to our new class. In this example, `_DEFAULTS` refers to a dictionary with the default hyperparameter values that we defined. `from_config_dict` is called to instantiate the transform given hyperparameters from the config. +```python + @classmethod + def from_config_dict(cls, config=None): + _config = {} if config is None else config + return ConcatAugment( + _config.get("rate", _DEFAULTS["rate"]), + _config.get("max_tokens", _DEFAULTS["max_tokens"]), + _config.get("attempts", _DEFAULTS["attempts"]), + ) +``` +We edit the instantiation function `__init__` to track hyperparameters and do any setup work. +```python + def __init__( + self, + rate=_DEFAULTS["rate"], + max_tokens=_DEFAULTS["max_tokens"], + attempts=_DEFAULTS["attempts"], + ): + self.rate, self.max_tokens, self.attempts = rate, max_tokens, attempts +``` +Lastly `__repr__` gives how the transform will be reported in an output log. +```python + def __repr__(self): + return ( + self.__class__.__name__ + + "(" + + ", ".join( + [ + f"rate={self.rate}", + f"max_tokens={self.max_tokens}", + f"attempts={self.attempts}", + ] + ) + + ")" + ) +``` + +### Step 3. Adding the transform logic +At this point, we are ready to implement the actual transform logic. The flow from here is different for each of the three transforms, so follow the path that is relevant to you. +### ...for feature transforms +The final step is implementing the `__call__` function, which applies the transform logic and **returns** the spectrogram with transform applied. This supports and should take exactly **two arguments**: +- `self` +- `x` (np.ndarray): the spectrogram for one source sample. (This is a positional argument, so you can use another parameter name like `spectrogram` instead of `x`.) + +For example, this is the `__call__` function for GlobalCMVN (cepstral mean and variance normalization). +```python + def __call__(self, x): + x = np.subtract(x, self.mean) + x = np.divide(x, self.std) + return x + +``` +### ...for waveform transforms +The final step is implementing the `__call__` function, which applies the transform logic. This supports and should take exactly **three arguments**: +- `self` +- `source` (numpy.ndarray or torch.Tensor): source audio 2d waveform (channels x length) +- `sample_rate` (optional, defaults to None): sample rate of `source` + +`__call__` **returns**: +- transformed audio waveform +- sample rate of transformed audio waveform + +For example, this is the `__call__` function for augmentations in the Noise Augmentation Suite. +```python + def __call__(self, source, sample_rate=None): + if np.random.random() > self.rate: + return source + + noise = self._get_noise( + source.shape, always_2d=True, use_sample_rate=sample_rate + ) + return self._mix(source, noise, rand_uniform(self.snr_min, self.snr_max)), sample_rate +``` + +### ...for dataset transforms +Dataset transforms are extremely flexible, and implementation involves directly integrating them into `fairseq/data/audio/speech_to_text_dataset.py` in transform-specific ways. +There are two basic components: (1) check whether or not this transform is part of this dataset instance using `self.dataset_transforms.has_transform(TRANSFORM_CLS)`, and (2) if so, get the transform using `self.dataset_transforms.get_transform(TRANSFORM_CLS)` & apply it. +Due to the case-by-case specificity, it is easier to demonstrate this by examples. + +#### Example: NoisyOverlapAugment +This transform requires access to multiple items within the same batch at once. + +**Logic**: We still use the transform classes to keep away the transform logic. For example, `__call__` of `NoisyOverlapAugment` class takes a list of source tokens for items in a mini-batch, applies noise/utterance as dictated by the transform, and returns the list of transformed source tokens for items in the mini-batch. + +```python + def __call__(self, sources): + for i, source in enumerate(sources): + if np.random.random() > self.rate: + continue + + pri = source.numpy() + + # ... some transform code omitted + + pri[s_source : s_source + l] = np.add( + pri[s_source : s_source + l], np.multiply(scl, sec[s_sec : s_sec + l]) + ) + sources[i] = torch.from_numpy(pri).float() + + return sources +``` + +**Integration**: The `collater` function for `SpeechToTextDataset` is responsible for preparing a mini-batch for training, so we integrate NOAug through adding a few lines to the top of this function: +```python +def collater( + self, samples: List[SpeechToTextDatasetItem], return_order: bool = False +) -> Dict: + if len(samples) == 0: + return {} + indices = torch.tensor([x.index for x in samples], dtype=torch.long) + + sources = [x.source for x in samples] + + # NOAUG INTEGRATION BLOCK + # (1) Check whether or not this transform is part of this dataset instance + has_NOAug = self.dataset_transforms.has_transform(NoisyOverlapAugment) + # (2) If so, get & apply the transform + if has_NOAug and self.cfg.use_audio_input: + NOAug = self.dataset_transforms.get_transform(NoisyOverlapAugment) + sources = NOAug(sources) + + frames = _collate_frames(sources, self.cfg.use_audio_input) + # sort samples by descending number of frames + n_frames = torch.tensor([x.size(0) for x in sources], dtype=torch.long) + n_frames, order = n_frames.sort(descending=True) + indices = indices.index_select(0, order) + frames = frames.index_select(0, order) + + # ... rest of function +``` + +#### Example: ConcatAugment +This transform requires access to another item within the dataset at once. + +**Logic**: We abstract the logic for picking indices to concatenate by adding a `find_indices` function to the `ConcatAugment` class, which takes one index in the dataset and finds a compatible second index to concatenate source and target tokens. +```python + def find_indices(self, index: int, n_frames: List[int], n_samples: int): + # skip conditions: application rate, max_tokens limit exceeded + if np.random.random() > self.rate: + return [index] + if self.max_tokens and n_frames[index] > self.max_tokens: + return [index] + + # pick second sample to concatenate + for _ in range(self.attempts): + index2 = np.random.randint(0, n_samples) + if index2 != index and ( + not self.max_tokens + or n_frames[index] + n_frames[index2] < self.max_tokens + ): + return [index, index2] + + return [index] +``` + +**Integration**: `SpeechToTextDataset` uses a custom `__getitem__(self, index)` function (called in the background when you write `dataset[i]`). We edited this function (as well as `_get_source_audio` and `get_tokenized_tgt_text`) to achieve the desired transform effect where accessing `dataset[i]` will return a `SpeechToTextDatasetItem` where `source=source[i]+source[j]` and `target=target[i]+target[j]`. +```python +def __getitem__(self, index: int) -> SpeechToTextDatasetItem: + + # CONCATAUGMENT INTEGRATION BLOCK + # (1) Check whether or not this transform is part of this dataset instance + has_concat = self.dataset_transforms.has_transform(ConcatAugment) + # (2) If so, get & apply the transform + if has_concat: + concat = self.dataset_transforms.get_transform(ConcatAugment) + indices = concat.find_indices(index, self.n_frames, self.n_samples) + + source = self._get_source_audio(indices if has_concat else index) + source = self.pack_frames(source) + + target = None + if self.tgt_texts is not None: + tokenized = self.get_tokenized_tgt_text(indices if has_concat else index) + target = self.tgt_dict.encode_line( + + # ... rest of function +``` diff --git a/data/fairseq/examples/speech_to_speech/docs/direct_s2st_discrete_units.md b/data/fairseq/examples/speech_to_speech/docs/direct_s2st_discrete_units.md new file mode 100644 index 0000000000000000000000000000000000000000..0c63ffee1cc4eff67cb3b761e4771dbf33f36e1a --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/docs/direct_s2st_discrete_units.md @@ -0,0 +1,181 @@ +# Direct speech-to-speech translation with discrete units + +We provide the implementation for speech-to-unit translation (S2UT) proposed in "[Direct speech-to-speech translation with discrete units (Lee et al. 2021)](https://arxiv.org/abs/2107.05604)" and also the transformer-based implementation of the speech-to-spectrogram translation (S2SPECT, or transformer-based [Translatotron](https://arxiv.org/abs/1904.06037)) baseline in the paper. + +## Pretrained Models + +### Unit-based HiFi-GAN Vocoder +Unit config | Unit size | Vocoder dataset | Model +|---|---|---|--- +[HuBERT Base, Librispeech](https://github.com/fairinternal/fairseq-py/tree/main/examples/hubert), layer 6 | 100 | [LJSpeech](https://keithito.com/LJ-Speech-Dataset/) | [ckpt](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/vocoder/code_hifigan/hubert_base_100_lj/g_00500000), [config](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/vocoder/code_hifigan/hubert_base_100_lj/config.json) + + +## Data preparation +### Target speech +0. (optional) To prepare S2S data from a speech-to-text translation (ST) dataset, see [fairseq-S^2](https://github.com/pytorch/fairseq/tree/main/examples/speech_synthesis) for pre-trained TTS models and instructions on how to train and decode TTS models. +1. Prepare two folders, `$SRC_AUDIO` and `$TGT_AUDIO`, with `${SPLIT}/${SAMPLE_ID}.wav` for source and target speech under each folder, separately. Note that for S2UT experiments, target audio sampling rate should be in 16,000 Hz, and for S2SPECT experiments, target audio sampling rate is recommended to be in 22,050 Hz. +2. To prepare target discrete units for S2UT model training, see [Generative Spoken Language Modeling (speech2unit)](https://github.com/pytorch/fairseq/tree/main/examples/textless_nlp/gslm/speech2unit) for pre-trained k-means models, checkpoints, and instructions on how to decode units from speech. Set the output target unit files (`--out_quantized_file_path`) as `${TGT_AUDIO}/${SPLIT}.txt`. In [Lee et al. 2021](https://arxiv.org/abs/2107.05604), we use 100 units from the sixth layer (`--layer 6`) of the HuBERT Base model. + +### Formatting data +**Speech-to-speech data** + +_S2UT_ + * Set `--reduce-unit` for training S2UT _reduced_ model + * Pre-trained vocoder and config (`$VOCODER_CKPT`, `$VOCODER_CFG`) can be downloaded from the **Pretrained Models** section. They are not required if `--eval-inference` is not going to be set during model training. +``` +# $SPLIT1, $SPLIT2, etc. are split names such as train, dev, test, etc. + +python examples/speech_to_speech/preprocessing/prep_s2ut_data.py \ + --source-dir $SRC_AUDIO --target-dir $TGT_AUDIO --data-split $SPLIT1 $SPLIT2 \ + --output-root $DATA_ROOT --reduce-unit \ + --vocoder-checkpoint $VOCODER_CKPT --vocoder-cfg $VOCODER_CFG +``` + +_S2SPECT_ +``` +# $SPLIT1, $SPLIT2, etc. are split names such as train, dev, test, etc. + +python examples/speech_to_speech/preprocessing/prep_s2spect_data.py \ + --source-dir $SRC_AUDIO --target-dir $TGT_AUDIO --data-split $SPLIT1 $SPLIT2 \ + --output-root $DATA_ROOT +``` + +**Multitask data** + * For each multitask `$TASK_NAME`, prepare `${DATA_ROOT}/${TASK_NAME}/${SPLIT}.tsv` files for each split following the format below: (Two tab separated columns. The sample_ids should match with the sample_ids for the speech-to-speech data in `${DATA_ROOT}/${SPLIT}.tsv`.) +``` +id tgt_text +sample_id_0 token1 token2 token3 ... +sample_id_1 token1 token2 token3 ... +... +``` + * For each multitask `$TASK_NAME`, prepare `${DATA_ROOT}/${TASK_NAME}/dict.txt`, a dictionary in fairseq format with all tokens for the targets for `$TASK_NAME`. + * Create `config_multitask.yaml`. Below is an example of the config used for S2UT _reduced_ with Fisher experiments including two encoder multitasks (`source_letter`, `target_letter`) and one decoder CTC task (`decoder_target_ctc`). +``` +source_letter: # $TASK_NAME + decoder_type: transformer + dict: ${DATA_ROOT}/source_letter/dict.txt + data: ${DATA_ROOT}/source_letter + encoder_layer: 6 + loss_weight: 8.0 +target_letter: + decoder_type: transformer + dict: ${DATA_ROOT}/target_letter/dict.txt + data: ${DATA_ROOT}/target_letter + encoder_layer: 8 + loss_weight: 8.0 +decoder_target_ctc: + decoder_type: ctc + dict: ${DATA_ROOT}/decoder_target_ctc/dict.txt + data: ${DATA_ROOT}/decoder_target_ctc + decoder_layer: 3 + loss_weight: 1.6 +``` + + +## Training + +**Speech-to-unit translation (S2UT)** + +Here's an example for training Fisher S2UT models with 100 discrete units as target: +``` +fairseq-train $DATA_ROOT \ + --config-yaml config.yaml --multitask-config-yaml config_multitask.yaml \ + --task speech_to_speech --target-is-code --target-code-size 100 --vocoder code_hifigan \ + --criterion speech_to_unit --label-smoothing 0.2 \ + --arch s2ut_transformer_fisher --share-decoder-input-output-embed \ + --dropout 0.1 --attention-dropout 0.1 --relu-dropout 0.1 \ + --train-subset train --valid-subset dev \ + --save-dir ${MODEL_DIR} \ + --lr 0.0005 --lr-scheduler inverse_sqrt --warmup-init-lr 1e-7 --warmup-updates 10000 \ + --optimizer adam --adam-betas "(0.9,0.98)" --clip-norm 10.0 \ + --max-update 400000 --max-tokens 20000 --max-target-positions 3000 --update-freq 4 \ + --seed 1 --fp16 --num-workers 8 +``` +* Adjust `--update-freq` accordingly for different #GPUs. In the above we set `--update-freq 4` to simulate training with 4 GPUs. +* Set `--n-frames-per-step 5` to train an S2UT _stacked_ system with reduction ratio r=5. (Use `$DATA_ROOT` prepared without `--reduce-unit`.) +* (optional) one can turn on tracking MCD loss during training for checkpoint selection by setting `--eval-inference --eval-args '{"beam": 1, "max_len_a": 1}' --best-checkpoint-metric mcd_loss`. It is recommended to sample a smaller subset as the validation set as MCD loss computation is time-consuming. + +**Speech-to-spectrogram translation (S2SPECT)** + +Here's an example for training Fisher S2SPECT models with reduction ratio r=5: +``` +fairseq-train $DATA_ROOT \ + --config-yaml config.yaml --multitask-config-yaml config_multitask.yaml \ + --task speech_to_speech --n-frames-per-step 5 \ + --criterion speech_to_spectrogram \ + --arch s2spect_transformer_fisher --decoder-normalize-before \ + --dropout 0.1 --attention-dropout 0.1 --relu-dropout 0.1 \ + --train-subset train --valid-subset dev \ + --save-dir ${MODEL_DIR} \ + --eval-inference --best-checkpoint-metric mcd_loss \ + --lr 0.0005 --lr-scheduler inverse_sqrt --warmup-init-lr 1e-7 --warmup-updates 10000 \ + --optimizer adam --adam-betas "(0.9,0.98)" --clip-norm 10.0 --weight-decay 1e-6 \ + --max-update 400000 --max-tokens 80000 --max-tokens-valid 30000 --required-batch-size-multiple 1 \ + --max-target-positions 3000 --update-freq 16 \ + --seed 1 --fp16 --num-workers 8 +``` +* Adjust `--update-freq` accordingly for different #GPUs. In the above we set `--update-freq 16` to simulate training with 16 GPUs. +* We recommend turning on MCD loss during training for the best checkpoint selection. + +**Unit-based HiFi-GAN vocoder** + +The vocoder is trained with the [speech-resynthesis repo](https://github.com/facebookresearch/speech-resynthesis). See [here](https://github.com/facebookresearch/speech-resynthesis/tree/main/examples/speech_to_speech_translation) for instructions on how to train the unit-based HiFi-GAN vocoder with duration prediction. The same vocoder can support waveform generation for both _reduced_ unit sequences (with `--dur-prediction` set during inference) and original unit sequences. + +## Inference + +**Speech-to-unit translation (S2UT)** + +1. Follow the same inference process as in [fairseq-S2T](https://github.com/pytorch/fairseq/tree/main/examples/speech_to_text) to generate unit sequences (`${RESULTS_PATH}/generate-${GEN_SUBSET}.txt`). +``` +fairseq-generate $DATA_ROOT \ + --config-yaml config.yaml --multitask-config-yaml config_multitask.yaml \ + --task speech_to_speech --target-is-code --target-code-size 100 --vocoder code_hifigan \ + --path $MODEL_DIR/checkpoint_best.pt --gen-subset $GEN_SUBSET \ + --max-tokens 50000 \ + --beam 10 --max-len-a 1 \ + --results-path ${RESULTS_PATH} +``` + * Set `--beam 1 --n-frames-per-step $r` for decoding with S2UT _stacked_ models. + +2. Convert unit sequences to waveform. +``` +grep "^D\-" ${RESULTS_PATH}/generate-${GEN_SUBSET}.txt | \ + sed 's/^D-//ig' | sort -nk1 | cut -f3 \ + > ${RESULTS_PATH}/generate-${GEN_SUBSET}.unit + +python examples/speech_to_speech/generate_waveform_from_code.py \ + --in-code-file ${RESULTS_PATH}/generate-${GEN_SUBSET}.unit \ + --vocoder $VOCODER_CKPT --vocoder-cfg $VOCODER_CFG \ + --results-path ${RESULTS_PATH} --dur-prediction +``` + * Set `--dur-prediction` for generating audio for S2UT _reduced_ models. + + +**Speech-to-spectrogram translation (S2SPECT)** + +Follow the same inference process as in [fairseq-S^2](https://github.com/pytorch/fairseq/tree/main/examples/speech_synthesis) to generate waveform. + +``` +# assume using a default Griffin-Lim vocoder + +python examples/speech_synthesis/generate_waveform.py $DATA_ROOT \ + --config-yaml config.yaml --multitask-config-yaml config_multitask.yaml \ + --task speech_to_speech --n-frames-per-step 5 \ + --path $MODEL_DIR/checkpoint_best.pt --gen-subset $GEN_SUBSET \ + --max-tokens 50000 \ + --results-path ${RESULTS_PATH} --dump-waveforms --output-sample-rate 16000 +``` + +In addition to using the default Griffin-Lim vocoder, one can also finetune a HiFi-GAN vocoder for the S2SPECT model by following the instructions in the [HiFi-GAN repo](https://github.com/jik876/hifi-gan). + +**Multitask decoding** + +Coming soon. + +## Evaluation + +To evaluate speech translation output, we first apply ASR on the speech output and then compute BLEU score betweent the ASR decoded text and the references using sacreBLEU. + +**En** +* ASR: We use the "[Wav2Vec 2.0 Large (LV-60) + Self Training / 960 hours / Libri-Light + Librispeech](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_vox_960h_pl.pt)" En ASR model open-sourced by the [wav2vec](https://github.com/pytorch/fairseq/tree/main/examples/wav2vec) project. See [instructions](https://github.com/pytorch/fairseq/tree/main/examples/wav2vec#evaluating-a-ctc-model) on how to run inference with a wav2vec-based ASR model. The model is also available on [Hugging Face](https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self). +* Text normalization: We use the text cleaner at [https://github.com/keithito/tacotron](https://github.com/keithito/tacotron) for pre-processing reference English text for ASR BLEU evaluation. diff --git a/data/fairseq/examples/speech_to_speech/docs/enhanced_direct_s2st_discrete_units.md b/data/fairseq/examples/speech_to_speech/docs/enhanced_direct_s2st_discrete_units.md new file mode 100644 index 0000000000000000000000000000000000000000..fbfa5dd16a760e8babe039cb74c1df8b644a89fe --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/docs/enhanced_direct_s2st_discrete_units.md @@ -0,0 +1,125 @@ +# Speech to speech translation (S2ST) + +We provide the implementation for speech-to-unit translation (S2UT) proposed in [Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (Popuri et al. 2022)](https://arxiv.org/abs/2204.02967) and the various pretrained models used. + +## Pretrained Models + +### Unit extraction + +We used the multilingual HuBERT model open sourced in [Textless S2ST with Real Data](textless_s2st_real_data.md) + +### Wav2vec 2.0 + +Language | Block type | Model size | Dataset | Model | +--- | --- | --- | --- | --- | +Es | Transformer | BASE | Voxpopuli | [ckpt](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/s2st_finetuning/w2v2/es/transformer_B.pt) | +Es | Transformer | LARGE | Voxpopuli | [ckpt](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/s2st_finetuning/w2v2/es/transformer_L.pt) | +Es | Conformer | LARGE | Voxpopuli | [ckpt](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/s2st_finetuning/w2v2/es/conformer_L.pt) | +En | Transformer | BASE | Librilight| [ckpt](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/s2st_finetuning/w2v2/en/transformer_B.pt) | +En | Conformer | LARGE | Librilight | [ckpt](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/s2st_finetuning/w2v2/en/conformer_L.pt) | + +### Unit mBART + +Unit size | Dataset | Unit config | Model | +--- | --- | --- | --- | +1000 | [Voxpopuli](https://aclanthology.org/2021.acl-long.80) En, Es unlabelled speech | [mbart_large](https://github.com/pytorch/fairseq/blob/f591cc94caa85098ccf125a4782f91125b6a086d/fairseq/models/bart/model.py#L368) |[ckpt](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/s2st_finetuning/unit_mBART/checkpoint.pt) | + +## Data preparation + +1. To prepare data for S2UT finetuning, follow the steps from [Direct S2ST with Discrete Units](./direct_s2st_discrete_units.md) and format the data in the _S2UT_ format. Note that we use 1000 units from the eleventh layer (`--layer 11`) of the multilingual hubert model linked above instead +2. Run + +``` +var="id\taudio\tn_frames\ttgt_text\ttgt_n_frames" +sed -i "1s/.*/$var/" ${SPLIT}.tsv +``` + +## Training + +**Speech-to-unit translation (S2UT)** + +Here's an example for finetuning S2UT models with 1000 discrete units as target. You can download the sample [config](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/s2st_finetuning/config.yaml) file and [vocabulary](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/s2st_finetuning/dict.txt) for Es-En from here: + +``` +fairseq-train $DATA_ROOT \ + --config-yaml config.yaml \ + --task speech_to_text --arch xm_transformer\ + --criterion l --label-smoothing 0.2 \ + --share-decoder-input-output-embed --adaptor-n-layers 1 --normalize\ + --dropout 0.1 --attention-dropout 0.1 --relu-dropout 0.1 \ + --train-subset train --valid-subset dev \ + --load-pretrained-decoder-from ${unit_mBART} --w2v-path ${wav2vec2.0} \ + --mask-prob 0.3 --mask-channel-length 32 --mask-channel-prob 0.25\ + --save-dir ${MODEL_DIR} --checkpoint-activations --encoder-proj \ + --lr 0.0005 --dropout 0.1 --attention-dropout 0.1 --lr-scheduler inverse_sqrt\ + --warmup-init-lr 1e-7 --warmup-updates 10000 \ + --optimizer adam --adam-betas "(0.9,0.98)" --clip-norm 10.0 \ + --max-update 20000 --max-tokens 4000 --max-tokens-valid 4000 --max-source-positions 4000 \ + --max-target-positions 4000 --update-freq 120 \ + --seed 1 --fp16 --num-workers 1 +``` + +* Adjust `--update-freq` accordingly for different #GPUs. In the above we set `--update-freq 15` to simulate training with 120 GPUs. +* In the above setting we finetune the model end to end, corresponding to the full setup in the paper. +* To apply LNA-E partial finetuning, add `--finetune-w2v-params layer_norm,self_attn` +* For LNA-D partial finetuning add `--finetune-decoder-params encoder_attn,layer_norm,self_attn`. To optionally freeze the encoder by k updates, use `--freeze-finetune-updates ${K}` +* For LNA-E,D partial finetuning add both the above options. + +**Unit-based HiFi-GAN vocoder** + +We apply the open-sourced unit-based HiFi-GAN vocoders to convert the predicted unit sequences to waveform. They are open sourced in [Textless S2ST with Real Data](textless_s2st_real_data.md) + +## Inference + +**Speech-to-unit translation (S2UT)** + +1. Follow the same inference process as in [fairseq-S2T](https://github.com/pytorch/fairseq/tree/main/examples/speech_to_text) to generate unit sequences (`${RESULTS_PATH}/generate-${GEN_SUBSET}.txt`). + +``` +fairseq-generate $DATA_ROOT \ + --config-yaml config.yaml \ + --task speech_to_text \ + --path $MODEL_DIR/checkpoint_best.pt --gen-subset $GEN_SUBSET \ + --max-tokens 10000 --max-source-positions 10000 --max-target-positions 10000\ + --beam 10 --max-len-a 1 --max-len-b 200 \ + --results-path ${RESULTS_PATH} +``` + +2. Convert unit sequences to waveform. + +``` +grep "^D\-" ${RESULTS_PATH}/generate-${GEN_SUBSET}.txt | \ + sed 's/^D-//ig' | sort -nk1 | cut -f3 \ + > ${RESULTS_PATH}/generate-${GEN_SUBSET}.unit + +python examples/speech_to_speech/generate_waveform_from_code.py \ + --in-code-file ${RESULTS_PATH}/generate-${GEN_SUBSET}.unit \ + --vocoder $VOCODER_CKPT --vocoder-cfg $VOCODER_CFG \ + --results-path ${RESULTS_PATH} --dur-prediction +``` + +## Evaluation + +To evaluate speech translation output, we first apply ASR on the speech output and then compute BLEU score betweent the ASR decoded text and the references using sacreBLEU. + +* Text normalization: We use the text cleaner at [https://github.com/keithito/tacotron](https://github.com/keithito/tacotron) for pre-processing reference English text for ASR BLEU evaluation. The text cleaner used for Spanish text normalization will be updated here shortly. +* En ASR: We use the "[Wav2Vec 2.0 Large (LV-60) + Self Training / 960 hours / Libri-Light + Librispeech](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_vox_960h_pl.pt)" En ASR model open-sourced by the [wav2vec](https://github.com/pytorch/fairseq/tree/main/examples/wav2vec) project. The model is also available on [Hugging Face](https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self). +* Es ASR: We use the [Wav2Vec2-Large-XLSR-53-Spanish](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) finetuned on spanish Common Voice Es ASR model open-sourced by Jonatasgrosman() on [Hugging Face](https://huggingface.co/jonatasgrosman/wav2vec2-large-xlsr-53-spanish). +* See [instructions](https://github.com/pytorch/fairseq/tree/main/examples/wav2vec#evaluating-a-ctc-model) on how to run inference with a wav2vec-based ASR model. + + +## Finetuned Model Checkpoints + +ID | En - Es | Es - En | +| --- | --- | --- | +**S2UT systems without pre-training** +S2UT with multitask | [checkpoint](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/s2st_finetuning/en_es//S2UT_w_multitask.pt) | [checkpoint](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/s2st_finetuning/es_en//S2UT_w_multitask.pt) | +**S2UT systems with model pre-training** +w2v2-L | [checkpoint](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/s2st_finetuning/en_es//w2v2_only.pt ) | [checkpoint](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/s2st_finetuning/es_en//w2v2_only.pt) | +w2v2-L + mBART (LNA-E) | [checkpoint](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/s2st_finetuning/en_es//w2v2_mbart_LNE.pt) | [checkpoint](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/s2st_finetuning/es_en//w2v2_mbart_LNE.pt) | +w2v2-L + mBART (LNA-D) | [checkpoint](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/s2st_finetuning/en_es//w2v2_mbart_LND.pt) | [checkpoint](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/s2st_finetuning/es_en//w2v2_mbart_LND.pt) | +w2v2-L + mBART (LNA-E,D) | [checkpoint](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/s2st_finetuning/en_es//w2v2_mbart_LNED.pt) | [checkpoint](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/s2st_finetuning/es_en//w2v2_mbart_LNED.pt) | +**S2UT systems with model pre-training and data augmentation** +w2v2-L + mBART (LNA-D) | [checkpoint](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/s2st_finetuning/en_es//w2v2_mbart_LND_w_ASR.pt) | [checkpoint](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/s2st_finetuning/es_en//w2v2_mbart_LND_w_ASR.pt) | + +Note: Some of the tasks use speech_to_text_sharded task which is yet to be open sourced. So make sure to override the task to speech_to_text to use those models. diff --git a/data/fairseq/examples/speech_to_speech/docs/textless_s2st_real_data.md b/data/fairseq/examples/speech_to_speech/docs/textless_s2st_real_data.md new file mode 100644 index 0000000000000000000000000000000000000000..ca6044be1ac9abf21cdcc91e1024ef64fb93cdcc --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/docs/textless_s2st_real_data.md @@ -0,0 +1,89 @@ +# Textless Speech-to-Speech Translation (S2ST) on Real Data + +We provide instructions and pre-trained models for the work "[Textless Speech-to-Speech Translation on Real Data (Lee et al. 2021)](https://arxiv.org/abs/2112.08352)". + +## Pre-trained Models + +### HuBERT +Model | Pretraining Data | Model | Quantizer +|---|---|---|--- +mHuBERT Base | [VoxPopuli](https://github.com/facebookresearch/voxpopuli) En, Es, Fr speech from the 100k subset | [download](https://dl.fbaipublicfiles.com/hubert/mhubert_base_vp_en_es_fr_it3.pt) | [L11 km1000](https://dl.fbaipublicfiles.com/hubert/mhubert_base_vp_en_es_fr_it3_L11_km1000.bin) + + +### Unit-based HiFi-GAN vocoder +Unit config | Unit size | Vocoder language | Dataset | Model +|---|---|---|---|--- +mHuBERT, layer 11 | 1000 | En | [LJSpeech](https://keithito.com/LJ-Speech-Dataset/) | [ckpt](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/vocoder/code_hifigan/mhubert_vp_en_es_fr_it3_400k_layer11_km1000_lj/g_00500000), [config](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/vocoder/code_hifigan/mhubert_vp_en_es_fr_it3_400k_layer11_km1000_lj/config.json) +mHuBERT, layer 11 | 1000 | Es | [CSS10](https://github.com/Kyubyong/css10) | [ckpt](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/vocoder/code_hifigan/mhubert_vp_en_es_fr_it3_400k_layer11_km1000_es_css10/g_00500000), [config](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/vocoder/code_hifigan/mhubert_vp_en_es_fr_it3_400k_layer11_km1000_es_css10/config.json) +mHuBERT, layer 11 | 1000 | Fr | [CSS10](https://github.com/Kyubyong/css10) | [ckpt](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/vocoder/code_hifigan/mhubert_vp_en_es_fr_it3_400k_layer11_km1000_fr_css10/g_00500000), [config](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/vocoder/code_hifigan/mhubert_vp_en_es_fr_it3_400k_layer11_km1000_fr_css10/config.json) + + +### Speech normalizer +Language | Training data | Target unit config | Model +|---|---|---|--- +En | 10 mins | mHuBERT, layer 11, km1000 | [download](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/speech_normalizer/en/en_10min.tar.gz) +En | 1 hr | mHuBERT, layer 11, km1000 | [download](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/speech_normalizer/en/en_1h.tar.gz) +En | 10 hrs | mHuBERT, layer 11, km1000 | [download](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/speech_normalizer/en/en_10h.tar.gz) +Es | 10 mins | mHuBERT, layer 11, km1000 | [download](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/speech_normalizer/es/es_10min.tar.gz) +Es | 1 hr | mHuBERT, layer 11, km1000 | [download](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/speech_normalizer/es/es_1h.tar.gz) +Es | 10 hrs | mHuBERT, layer 11, km1000 | [download](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/speech_normalizer/es/es_10h.tar.gz) +Fr | 10 mins | mHuBERT, layer 11, km1000 | [download](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/speech_normalizer/fr/fr_10min.tar.gz) +Fr | 1 hr | mHuBERT, layer 11, km1000 | [download](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/speech_normalizer/fr/fr_1h.tar.gz) +Fr | 10 hrs | mHuBERT, layer 11, km1000 | [download](https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/speech_normalizer/fr/fr_10h.tar.gz) + +* Refer to the paper for the details of the training data. + +## Inference with Pre-trained Models + +### Speech normalizer +1. Download the pre-trained models, including the dictionary, to `DATA_DIR`. +2. Format the audio data. +```bash +# AUDIO_EXT: audio extension, e.g. wav, flac, etc. +# Assume all audio files are at ${AUDIO_DIR}/*.${AUDIO_EXT} + +python examples/speech_to_speech/preprocessing/prep_sn_data.py \ + --audio-dir ${AUDIO_DIR} --ext ${AUIDO_EXT} \ + --data-name ${GEN_SUBSET} --output-dir ${DATA_DIR} \ + --for-inference +``` + +3. Run the speech normalizer and post-process the output. +```bash +mkdir -p ${RESULTS_PATH} + +python examples/speech_recognition/new/infer.py \ + --config-dir examples/hubert/config/decode/ \ + --config-name infer_viterbi \ + task.data=${DATA_DIR} \ + task.normalize=false \ + common_eval.results_path=${RESULTS_PATH}/log \ + common_eval.path=${DATA_DIR}/checkpoint_best.pt \ + dataset.gen_subset=${GEN_SUBSET} \ + '+task.labels=["unit"]' \ + +decoding.results_path=${RESULTS_PATH} \ + common_eval.post_process=none \ + +dataset.batch_size=1 \ + common_eval.quiet=True + +# Post-process and generate output at ${RESULTS_PATH}/${GEN_SUBSET}.txt +python examples/speech_to_speech/preprocessing/prep_sn_output_data.py \ + --in-unit ${RESULTS_PATH}/hypo.units \ + --in-audio ${DATA_DIR}/${GEN_SUBSET}.tsv \ + --output-root ${RESULTS_PATH} +``` + + +### Unit-to-waveform conversion with unit vocoder +The pre-trained vocoders can support generating audio for both full unit sequences and reduced unit sequences (i.e. duplicating consecutive units removed). Set `--dur-prediction` for generating audio with reduced unit sequences. +```bash +# IN_CODE_FILE contains one unit sequence per line. Units are separated by space. + +python examples/speech_to_speech/generate_waveform_from_code.py \ + --in-code-file ${IN_CODE_FILE} \ + --vocoder ${VOCODER_CKPT} --vocoder-cfg ${VOCODER_CFG} \ + --results-path ${RESULTS_PATH} --dur-prediction +``` + +## Training new models +To be updated. diff --git a/data/fairseq/examples/speech_to_speech/generate_waveform_from_code.py b/data/fairseq/examples/speech_to_speech/generate_waveform_from_code.py new file mode 100644 index 0000000000000000000000000000000000000000..82aa7acfb82ed6914cd04411d041503ac828ee30 --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/generate_waveform_from_code.py @@ -0,0 +1,116 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import argparse +import json +import logging +from pathlib import Path +import random +import soundfile as sf +import torch + +from tqdm import tqdm + +from fairseq import utils +from fairseq.models.text_to_speech.vocoder import CodeHiFiGANVocoder + + +logging.basicConfig() +logging.root.setLevel(logging.INFO) +logging.basicConfig(level=logging.INFO) +logger = logging.getLogger(__name__) + + +def dump_result(args, sample_id, pred_wav, suffix=""): + sf.write( + f"{args.results_path}/{sample_id}{suffix}_pred.wav", + pred_wav.detach().cpu().numpy(), + 16000, + ) + + +def load_code(in_file): + with open(in_file) as f: + out = [list(map(int, line.strip().split())) for line in f] + return out + + +def main(args): + logger.info(args) + + use_cuda = torch.cuda.is_available() and not args.cpu + + with open(args.vocoder_cfg) as f: + vocoder_cfg = json.load(f) + vocoder = CodeHiFiGANVocoder(args.vocoder, vocoder_cfg) + if use_cuda: + vocoder = vocoder.cuda() + + multispkr = vocoder.model.multispkr + if multispkr: + logger.info("multi-speaker vocoder") + num_speakers = vocoder_cfg.get( + "num_speakers", 200 + ) # following the default in codehifigan to set to 200 + assert ( + args.speaker_id < num_speakers + ), f"invalid --speaker-id ({args.speaker_id}) with total #speakers = {num_speakers}" + + data = load_code(args.in_code_file) + Path(args.results_path).mkdir(exist_ok=True, parents=True) + for i, d in tqdm(enumerate(data), total=len(data)): + x = { + "code": torch.LongTensor(d).view(1, -1), + } + suffix = "" + if multispkr: + spk = ( + random.randint(0, num_speakers - 1) + if args.speaker_id == -1 + else args.speaker_id + ) + suffix = f"_spk{spk}" + x["spkr"] = torch.LongTensor([spk]).view(1, 1) + + x = utils.move_to_cuda(x) if use_cuda else x + wav = vocoder(x, args.dur_prediction) + dump_result(args, i, wav, suffix=suffix) + + +def cli_main(): + parser = argparse.ArgumentParser() + parser.add_argument( + "--in-code-file", type=str, required=True, help="one unit sequence per line" + ) + parser.add_argument( + "--vocoder", type=str, required=True, help="path to the CodeHiFiGAN vocoder" + ) + parser.add_argument( + "--vocoder-cfg", + type=str, + required=True, + help="path to the CodeHiFiGAN vocoder config", + ) + parser.add_argument("--results-path", type=str, required=True) + parser.add_argument( + "--dur-prediction", + action="store_true", + help="enable duration prediction (for reduced/unique code sequences)", + ) + parser.add_argument( + "--speaker-id", + type=int, + default=-1, + help="Speaker id (for vocoder that supports multispeaker). Set to -1 to randomly sample speakers.", + ) + parser.add_argument("--cpu", action="store_true", help="run on CPU") + + args = parser.parse_args() + + main(args) + + +if __name__ == "__main__": + cli_main() diff --git a/data/fairseq/examples/speech_to_speech/preprocessing/__init__.py b/data/fairseq/examples/speech_to_speech/preprocessing/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..6264236915a7269a4d920ee8213004374dd86a9a --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/preprocessing/__init__.py @@ -0,0 +1,4 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. diff --git a/data/fairseq/examples/speech_to_speech/preprocessing/data_utils.py b/data/fairseq/examples/speech_to_speech/preprocessing/data_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..a83a67f9549e79792e78d013a1dabccd78475185 --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/preprocessing/data_utils.py @@ -0,0 +1,88 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +from pathlib import Path +from typing import List, Optional + +from examples.speech_to_text.data_utils import S2TDataConfigWriter + + +def gen_config_yaml( + manifest_root: Path, + yaml_filename: str = "config.yaml", + specaugment_policy: Optional[str] = "lb", + feature_transform: Optional[List[str]] = None, + input_channels: Optional[int] = 1, + input_feat_per_channel: Optional[int] = 80, + audio_root: str = "", + vocoder_type: Optional[str] = None, + vocoder_checkpoint: Optional[str] = None, + vocoder_cfg: Optional[str] = None, + extra=None, +): + manifest_root = manifest_root.absolute() + writer = S2TDataConfigWriter(manifest_root / yaml_filename) + + if input_channels is not None: + writer.set_input_channels(input_channels) + if input_feat_per_channel is not None: + writer.set_input_feat_per_channel(input_feat_per_channel) + specaugment_setters = { + "lb": writer.set_specaugment_lb_policy, + "ld": writer.set_specaugment_ld_policy, + "sm": writer.set_specaugment_sm_policy, + "ss": writer.set_specaugment_ss_policy, + } + specaugment_setter = specaugment_setters.get(specaugment_policy, None) + if specaugment_setter is not None: + specaugment_setter() + + if feature_transform is None: + feature_transform = [] + else: + writer.set_feature_transforms("*", feature_transform) + + if specaugment_policy is not None: + writer.set_feature_transforms("_train", feature_transform + ["specaugment"]) + + if len(audio_root) > 0: + writer.set_audio_root(audio_root) + + if ( + vocoder_type is not None + and vocoder_checkpoint is not None + and vocoder_cfg is not None + ): + writer.set_extra( + { + "vocoder": { + "type": vocoder_type, + "config": vocoder_cfg, + "checkpoint": vocoder_checkpoint, + } + } + ) + + if extra is not None: + writer.set_extra(extra) + writer.flush() + + +def load_units(in_file): + out = {} + with open(in_file) as f: + for line in f: + sample_id, units = line.strip().split("|", 1) + out[sample_id] = units.split() + + return out + + +def process_units(units, reduce=False): + if not reduce: + return units + + out = [u for i, u in enumerate(units) if i == 0 or u != units[i - 1]] + return out diff --git a/data/fairseq/examples/speech_to_speech/preprocessing/prep_s2spect_data.py b/data/fairseq/examples/speech_to_speech/preprocessing/prep_s2spect_data.py new file mode 100644 index 0000000000000000000000000000000000000000..2748b37aef71d04451febd612a4fee9bf9c8a91a --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/preprocessing/prep_s2spect_data.py @@ -0,0 +1,169 @@ +#!/usr/bin/env python3 +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import argparse +import logging +import os +from pathlib import Path +import shutil +import torchaudio + +import soundfile as sf +from tqdm import tqdm +import pandas as pd + +from examples.speech_synthesis.data_utils import extract_logmel_spectrogram +from examples.speech_to_speech.preprocessing.data_utils import gen_config_yaml +from examples.speech_to_text.data_utils import create_zip, get_zip_manifest, save_df_to_tsv +from fairseq.data.audio.audio_utils import convert_waveform + + +logger = logging.getLogger(__name__) + +MANIFEST_COLUMNS = ["id", "src_audio", "src_n_frames", "tgt_audio", "tgt_n_frames"] + + +def prepare_target_data(args, tgt_audios): + feature_name = "logmelspec80" + zip_path = args.output_root / f"{feature_name}.zip" + if zip_path.exists(): + print(f"{zip_path} exists.") + return zip_path + + feature_root = args.output_root / feature_name + feature_root.mkdir(exist_ok=True) + + print("Extracting Mel spectrogram features...") + for tgt_audio in tqdm(tgt_audios): + sample_id = tgt_audio.stem + waveform, sample_rate = torchaudio.load(tgt_audio.as_posix()) + waveform, sample_rate = convert_waveform( + waveform, sample_rate, normalize_volume=args.normalize_volume, + to_sample_rate=args.sample_rate + ) + extract_logmel_spectrogram( + waveform, sample_rate, feature_root / f"{sample_id}.npy", + win_length=args.win_length, hop_length=args.hop_length, + n_fft=args.n_fft, n_mels=args.n_mels, f_min=args.f_min, + f_max=args.f_max + ) + print("ZIPing features...") + create_zip(feature_root, zip_path) + shutil.rmtree(feature_root) + + return zip_path + + +def process(args): + os.makedirs(args.output_root, exist_ok=True) + + manifest = {} + tgt_audios = [] + for split in args.data_split: + print(f"Processing {split}...") + + manifest[split] = {c: [] for c in MANIFEST_COLUMNS} + missing_tgt_audios = [] + src_audios = list(args.source_dir.glob(f"{split}/*.wav")) + for src_audio in tqdm(src_audios): + sample_id = src_audio.stem + + tgt_audio = args.target_dir / split / f"{sample_id}.wav" + if not tgt_audio.is_file(): + missing_tgt_audios.append(sample_id) + continue + + tgt_audios.append(tgt_audio) + + src_n_frames = sf.info(src_audio.as_posix()).frames + manifest[split]["id"].append(sample_id) + manifest[split]["src_audio"].append(src_audio.as_posix()) + manifest[split]["src_n_frames"].append( + src_n_frames // 160 + ) # estimation of 10-ms frame for 16kHz audio + + print(f"Processed {len(manifest[split]['id'])} samples") + if len(missing_tgt_audios) > 0: + print( + f"{len(missing_tgt_audios)} with missing target data (first 3 examples: {', '.join(missing_tgt_audios[:3])})" + ) + + # Extract features and pack features into ZIP + zip_path = prepare_target_data(args, tgt_audios) + + print("Fetching ZIP manifest...") + tgt_audio_paths, tgt_audio_lengths = get_zip_manifest(zip_path) + + print("Generating manifest...") + for split in args.data_split: + print(f"Processing {split}...") + + for sample_id in tqdm(manifest[split]["id"]): + manifest[split]["tgt_audio"].append(tgt_audio_paths[sample_id]) + manifest[split]["tgt_n_frames"].append(tgt_audio_lengths[sample_id]) + + out_manifest = args.output_root / f"{split}.tsv" + print(f"Writing manifest to {out_manifest}...") + save_df_to_tsv(pd.DataFrame.from_dict(manifest[split]), out_manifest) + + # Generate config YAML + win_len_t = args.win_length / args.sample_rate + hop_len_t = args.hop_length / args.sample_rate + extra = { + "features": { + "type": "spectrogram+melscale+log", + "sample_rate": args.sample_rate, + "eps": 1e-5, "n_mels": args.n_mels, "n_fft": args.n_fft, + "window_fn": "hann", "win_length": args.win_length, + "hop_length": args.hop_length, + "win_len_t": win_len_t, "hop_len_t": hop_len_t, + "f_min": args.f_min, "f_max": args.f_max, + "n_stft": args.n_fft // 2 + 1 + } + } + gen_config_yaml( + args.output_root, + audio_root=args.output_root.as_posix(), + specaugment_policy="lb", + feature_transform=["utterance_cmvn", "delta_deltas"], + extra=extra, + ) + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument( + "--source-dir", required=True, type=Path, help="source audio directory" + ) + parser.add_argument( + "--target-dir", required=True, type=Path, help="target audio directory" + ) + parser.add_argument( + "--data-split", + default=["train", "valid", "test"], + nargs="+", + help="data split names", + ) + parser.add_argument( + "--output-root", required=True, type=Path, help="output directory" + ) + # target feature related + parser.add_argument("--win-length", type=int, default=1024) + parser.add_argument("--hop-length", type=int, default=256) + parser.add_argument("--n-fft", type=int, default=1024) + parser.add_argument("--n-mels", type=int, default=80) + parser.add_argument("--f-min", type=int, default=20) + parser.add_argument("--f-max", type=int, default=8000) + parser.add_argument("--sample-rate", type=int, default=22050) + parser.add_argument("--normalize-volume", "-n", action="store_true") + + args = parser.parse_args() + + process(args) + + +if __name__ == "__main__": + main() diff --git a/data/fairseq/examples/speech_to_speech/preprocessing/prep_s2ut_data.py b/data/fairseq/examples/speech_to_speech/preprocessing/prep_s2ut_data.py new file mode 100644 index 0000000000000000000000000000000000000000..c97c0fe9be01fac1199b24f0b20b16c526494ff2 --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/preprocessing/prep_s2ut_data.py @@ -0,0 +1,114 @@ +#!/usr/bin/env python3 +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import argparse +import logging +from pathlib import Path + +import soundfile as sf +from tqdm import tqdm +import pandas as pd + +from examples.speech_to_speech.preprocessing.data_utils import ( + gen_config_yaml, + load_units, + process_units, +) +from examples.speech_to_text.data_utils import save_df_to_tsv + +logger = logging.getLogger(__name__) + +MANIFEST_COLUMNS = ["id", "src_audio", "src_n_frames", "tgt_audio", "tgt_n_frames"] + + +def process(args): + args.output_root.mkdir(exist_ok=True) + + print("Generating manifest...") + for split in args.data_split: + print(f"Processing {split}") + + # load target units + target_unit_data = load_units(args.target_dir / f"{split}.txt") + + manifest = {c: [] for c in MANIFEST_COLUMNS} + missing_tgt_audios = [] + src_audios = list(args.source_dir.glob(f"{split}/*.wav")) + for src_audio in tqdm(src_audios): + sample_id = src_audio.stem + + if sample_id not in target_unit_data: + missing_tgt_audios.append(sample_id) + continue + + src_n_frames = sf.info(src_audio.as_posix()).frames + manifest["id"].append(sample_id) + manifest["src_audio"].append(src_audio.as_posix()) + manifest["src_n_frames"].append( + src_n_frames // 160 + ) # estimation of 10-ms frame for 16kHz audio + + target_units = process_units(target_unit_data[sample_id], args.reduce_unit) + manifest["tgt_audio"].append(" ".join(target_units)) + manifest["tgt_n_frames"].append(len(target_units)) + + print(f"Processed {len(manifest['id'])} samples") + if len(missing_tgt_audios) > 0: + print( + f"{len(missing_tgt_audios)} with missing target data (first 3 examples: {', '.join(missing_tgt_audios[:3])})" + ) + + out_manifest = args.output_root / f"{split}.tsv" + print(f"Writing manifest to {out_manifest}...") + save_df_to_tsv(pd.DataFrame.from_dict(manifest), out_manifest) + + # Generate config YAML + gen_config_yaml( + args.output_root, + specaugment_policy="lb", + feature_transform=["utterance_cmvn"], + vocoder_type="code_hifigan", + vocoder_checkpoint=args.vocoder_checkpoint, + vocoder_cfg=args.vocoder_cfg, + ) + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument( + "--source-dir", required=True, type=Path, help="source audio directory" + ) + parser.add_argument( + "--target-dir", required=True, type=Path, help="target audio directory" + ) + parser.add_argument( + "--data-split", + default=["train", "valid", "test"], + nargs="+", + help="data split names", + ) + parser.add_argument( + "--output-root", required=True, type=Path, help="output directory" + ) + parser.add_argument( + "--reduce-unit", + action="store_true", + help="reduce a target unit sequence to a unique unit sequence, i.e. '1 1 1 2 2' -> '1 2'", + ) + parser.add_argument( + "--vocoder-checkpoint", default=None, type=str, help="vocoder checkpoint" + ) + parser.add_argument( + "--vocoder-cfg", default=None, type=str, help="vocoder config file" + ) + + args = parser.parse_args() + + process(args) + + +if __name__ == "__main__": + main() diff --git a/data/fairseq/examples/speech_to_speech/preprocessing/prep_sn_data.py b/data/fairseq/examples/speech_to_speech/preprocessing/prep_sn_data.py new file mode 100644 index 0000000000000000000000000000000000000000..ea94175634204075d645af2c35720c76c8924253 --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/preprocessing/prep_sn_data.py @@ -0,0 +1,88 @@ +#!/usr/bin/env python3 +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. +# +# Adapted from examples/wav2vec/wav2vec_manifest.py +""" +Data preparation for the speech normalizer +""" + +import argparse +import glob +import os + +import soundfile + +from examples.speech_to_speech.preprocessing.data_utils import load_units, process_units + + +def process(args): + assert ( + args.for_inference or args.target_unit is not None + ), "missing --target-unit or --for-inference" + + if not os.path.exists(args.output_dir): + os.makedirs(args.output_dir) + + dir_path = os.path.realpath(args.audio_dir) + search_path = os.path.join(dir_path, "**/*." + args.ext) + + if args.target_unit: + unit_data = load_units(args.target_unit) + + with open(os.path.join(args.output_dir, f"{args.data_name}.tsv"), "w") as o_t, open( + os.path.join(args.output_dir, f"{args.data_name}.unit"), "w" + ) as o_u: + print(dir_path, file=o_t) + for fname in glob.iglob(search_path, recursive=True): + file_path = os.path.realpath(fname) + frames = soundfile.info(fname).frames + print( + "{}\t{}".format(os.path.relpath(file_path, dir_path), frames), file=o_t + ) + + if args.for_inference: + print("0", file=o_u) + else: + sample_id = os.path.basename(file_path)[: -len(args.ext) - 1] + assert ( + sample_id in unit_data + ), f'{fname} does not have unit data in {args.target_unit}. Expecting sample_id "{sample_id}".' + target_units = process_units(unit_data[sample_id], reduce=True) + print(" ".join(target_units), file=o_u) + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--audio-dir", required=True, type=str, help="audio directory") + parser.add_argument("--ext", default="flac", type=str, help="audio extension") + parser.add_argument( + "--data-name", + required=True, + type=str, + help="dataset name", + ) + parser.add_argument( + "--output-dir", required=True, type=str, help="output directory" + ) + parser.add_argument( + "--for-inference", + action="store_true", + help="set this if preparing data for running inference with a speech normalizer", + ) + parser.add_argument( + "--target-unit", + default=None, + type=str, + help="a file containing unit sequences in the format: sample_id|u1 u2 ...", + ) + + args = parser.parse_args() + + process(args) + + +if __name__ == "__main__": + main() diff --git a/data/fairseq/examples/speech_to_speech/preprocessing/prep_sn_output_data.py b/data/fairseq/examples/speech_to_speech/preprocessing/prep_sn_output_data.py new file mode 100644 index 0000000000000000000000000000000000000000..06991343bd7ce9a3087a016d7e4bd04ca3a1fb97 --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/preprocessing/prep_sn_output_data.py @@ -0,0 +1,58 @@ +#!/usr/bin/env python3 +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import argparse +from pathlib import Path + +from tqdm import tqdm + + +def process(args): + args.output_root.mkdir(exist_ok=True) + + # load units + units = {} + with open(args.in_unit) as f: + for line in f: + unit_seq, utt_id = line.strip().rsplit(" ", 1) + utt_id = int(utt_id[6:-1]) # remove "(None-" + units[utt_id] = unit_seq + + with open(args.in_audio) as f, open( + args.output_root / f"{args.in_audio.stem}.txt", "w" + ) as o: + f.readline() + for i, line in enumerate(tqdm(f.readlines())): + audio, _ = line.strip().split("\t", 1) + sample_id = Path(audio).stem + o.write(f"{sample_id}|{units[i]}\n") + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument( + "--in-unit", + required=True, + type=Path, + help="unit file (output from the speech normalizer)", + ) + parser.add_argument( + "--in-audio", + required=True, + type=Path, + help="tsv file (input to the normalizer)", + ) + parser.add_argument( + "--output-root", required=True, type=Path, help="output directory" + ) + + args = parser.parse_args() + + process(args) + + +if __name__ == "__main__": + main() diff --git a/data/fairseq/examples/speech_to_speech/unity/__init__.py b/data/fairseq/examples/speech_to_speech/unity/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..349db7c65e51363d75ab6386188b4eef48ec89d0 --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/unity/__init__.py @@ -0,0 +1,7 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +from . import sequence_generator # noqa +from . import sequence_generator_multi_decoder # noqa diff --git a/data/fairseq/examples/speech_to_speech/unity/sequence_generator.py b/data/fairseq/examples/speech_to_speech/unity/sequence_generator.py new file mode 100644 index 0000000000000000000000000000000000000000..c482098feb88b1b055ffef86bb5dda4e0f5b28b0 --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/unity/sequence_generator.py @@ -0,0 +1,626 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import math +import sys +from typing import Dict, List, Optional + +import torch +from torch import Tensor + +from fairseq.sequence_generator import EnsembleModel as EnsembleModelBase +from fairseq.sequence_generator import SequenceGenerator as SequenceGeneratorBase + + +class SequenceGenerator(SequenceGeneratorBase): + def __init__( + self, + models, + tgt_dict, + beam_size=1, + max_len_a=0, + max_len_b=200, + max_len=0, + min_len=1, + normalize_scores=True, + len_penalty=1.0, + unk_penalty=0.0, + temperature=1.0, + match_source_len=False, + no_repeat_ngram_size=0, + search_strategy=None, + eos=None, + symbols_to_strip_from_output=None, + lm_model=None, + lm_weight=1.0, + tokens_to_suppress=(), + ): + """Generates translations of a given source sentence. + + Args: + models (List[~fairseq.models.FairseqModel]): ensemble of models, + currently support fairseq.models.TransformerModel for scripting + beam_size (int, optional): beam width (default: 1) + max_len_a/b (int, optional): generate sequences of maximum length + ax + b, where x is the source length + max_len (int, optional): the maximum length of the generated output + (not including end-of-sentence) + min_len (int, optional): the minimum length of the generated output + (not including end-of-sentence) + normalize_scores (bool, optional): normalize scores by the length + of the output (default: True) + len_penalty (float, optional): length penalty, where <1.0 favors + shorter, >1.0 favors longer sentences (default: 1.0) + unk_penalty (float, optional): unknown word penalty, where <0 + produces more unks, >0 produces fewer (default: 0.0) + temperature (float, optional): temperature, where values + >1.0 produce more uniform samples and values <1.0 produce + sharper samples (default: 1.0) + match_source_len (bool, optional): outputs should match the source + length (default: False) + """ + super().__init__( + models=models, + tgt_dict=tgt_dict, + beam_size=beam_size, + max_len_a=max_len_a, + max_len_b=max_len_b, + max_len=max_len, + min_len=min_len, + normalize_scores=normalize_scores, + len_penalty=len_penalty, + unk_penalty=unk_penalty, + temperature=temperature, + match_source_len=match_source_len, + no_repeat_ngram_size=no_repeat_ngram_size, + search_strategy=search_strategy, + eos=eos, + symbols_to_strip_from_output=symbols_to_strip_from_output, + lm_model=lm_model, + lm_weight=lm_weight, + tokens_to_suppress=tokens_to_suppress, + ) + + if isinstance(models, EnsembleModel): + self.model = models + else: + self.model = EnsembleModel(models) + + self.model.set_decoder_beam_size(self.beam_size) + self.model.eval() + + def _generate( + self, + sample: Dict[str, Dict[str, Tensor]], + prefix_tokens: Optional[Tensor] = None, + constraints: Optional[Tensor] = None, + bos_token: Optional[int] = None, + ): + net_input = sample["net_input"] + + if "src_tokens" in net_input: + src_tokens = net_input["src_tokens"] + # length of the source text being the character length except EndOfSentence and pad + # if src_lengths exists in net_input (speech_to_text dataset case), then use it + if "src_lengths" in net_input: + src_lengths = net_input["src_lengths"] + else: + src_lengths = ( + (src_tokens.ne(self.eos) & src_tokens.ne(self.pad)) + .long() + .sum(dim=1) + ) + elif "source" in net_input: + src_tokens = net_input["source"] + src_lengths = ( + net_input["padding_mask"].size(-1) - net_input["padding_mask"].sum(-1) + if net_input["padding_mask"] is not None + else torch.tensor(src_tokens.size(-1)).to(src_tokens) + ) + elif "features" in net_input: + src_tokens = net_input["features"] + src_lengths = ( + net_input["padding_mask"].size(-1) - net_input["padding_mask"].sum(-1) + if net_input["padding_mask"] is not None + else torch.tensor(src_tokens.size(-1)).to(src_tokens) + ) + else: + raise Exception( + "expected src_tokens or source in net input. input keys: " + + str(net_input.keys()) + ) + + if constraints is not None and not self.search.supports_constraints: + raise NotImplementedError( + "Target-side constraints were provided, but search method doesn't support them" + ) + + # Initialize constraints, when active + self.search.init_constraints(constraints, self.beam_size) + + # compute the encoder output for each beam + with torch.autograd.profiler.record_function("EnsembleModel: forward_encoder"): + encoder_outs = self.model.forward_encoder(net_input) + + finalized = self.generate_decoder( + encoder_outs, + src_tokens, + src_lengths, + sample, + prefix_tokens, + constraints, + bos_token, + ) + return finalized + + def generate_decoder( + self, + encoder_outs, + src_tokens, + src_lengths, + sample: Dict[str, Dict[str, Tensor]], + prefix_tokens: Optional[Tensor] = None, + constraints: Optional[Tensor] = None, + bos_token: Optional[int] = None, + aux_task_name="", + encoder_outs_aug: Optional[ + Tensor + ] = None, # an additional/augmented encoder_outs + ): + incremental_states = torch.jit.annotate( + List[Dict[str, Dict[str, Optional[Tensor]]]], + [ + torch.jit.annotate(Dict[str, Dict[str, Optional[Tensor]]], {}) + for i in range(self.model.models_size) + ], + ) + + # bsz: total number of sentences in beam + # Note that src_tokens may have more than 2 dimensions (i.e. audio features) + bsz, src_len = src_tokens.size()[:2] + beam_size = self.beam_size + + decoder_name = f"{aux_task_name}_decoder" if aux_task_name else "decoder" + + max_len: int = -1 + if self.match_source_len: + max_len = src_lengths.max().item() + else: + max_len = min( + int(self.max_len_a * src_len + self.max_len_b), + self.max_len - 1, + ) + assert ( + self.min_len <= max_len + ), "min_len cannot be larger than max_len, please adjust these!" + + # placeholder of indices for bsz * beam_size to hold tokens and accumulative scores + new_order = torch.arange(bsz).view(-1, 1).repeat(1, beam_size).view(-1) + new_order = new_order.to(src_tokens.device).long() + encoder_outs = self.model.reorder_encoder_out(encoder_outs, new_order) + # ensure encoder_outs is a List. + assert encoder_outs is not None + if encoder_outs_aug is not None: + encoder_outs_aug = self.model.reorder_encoder_out( + encoder_outs_aug, new_order + ) + + # initialize buffers + scores = ( + torch.zeros(bsz * beam_size, max_len + 1).to(src_tokens).float() + ) # +1 for eos; pad is never chosen for scoring + tokens = ( + torch.zeros(bsz * beam_size, max_len + 2) + .to(src_tokens) + .long() + .fill_(self.pad) + ) # +2 for eos and pad + tokens[:, 0] = self.eos if bos_token is None else bos_token + attn: Optional[Tensor] = None + + # A list that indicates candidates that should be ignored. + # For example, suppose we're sampling and have already finalized 2/5 + # samples. Then cands_to_ignore would mark 2 positions as being ignored, + # so that we only finalize the remaining 3 samples. + cands_to_ignore = ( + torch.zeros(bsz, beam_size).to(src_tokens).eq(-1) + ) # forward and backward-compatible False mask + + # list of completed sentences + finalized = torch.jit.annotate( + List[List[Dict[str, Tensor]]], + [torch.jit.annotate(List[Dict[str, Tensor]], []) for i in range(bsz)], + ) # contains lists of dictionaries of infomation about the hypothesis being finalized at each step + + # a boolean array indicating if the sentence at the index is finished or not + finished = [False for i in range(bsz)] + num_remaining_sent = bsz # number of sentences remaining + + # number of candidate hypos per step + cand_size = 2 * beam_size # 2 x beam size in case half are EOS + + # offset arrays for converting between different indexing schemes + bbsz_offsets = ( + (torch.arange(0, bsz) * beam_size) + .unsqueeze(1) + .type_as(tokens) + .to(src_tokens.device) + ) + cand_offsets = torch.arange(0, cand_size).type_as(tokens).to(src_tokens.device) + + reorder_state: Optional[Tensor] = None + batch_idxs: Optional[Tensor] = None + + original_batch_idxs: Optional[Tensor] = None + if "id" in sample and isinstance(sample["id"], Tensor): + original_batch_idxs = sample["id"] + else: + original_batch_idxs = torch.arange(0, bsz).type_as(tokens) + + for step in range(max_len + 1): # one extra step for EOS marker + # reorder decoder internal states based on the prev choice of beams + if reorder_state is not None: + if batch_idxs is not None: + # update beam indices to take into account removed sentences + corr = batch_idxs - torch.arange(batch_idxs.numel()).type_as( + batch_idxs + ) + reorder_state.view(-1, beam_size).add_( + corr.unsqueeze(-1) * beam_size + ) + original_batch_idxs = original_batch_idxs[batch_idxs] + self.model.reorder_incremental_state( + incremental_states, reorder_state, decoder_name + ) + encoder_outs = self.model.reorder_encoder_out( + encoder_outs, reorder_state + ) + if encoder_outs_aug is not None: + encoder_outs_aug = self.model.reorder_encoder_out( + encoder_outs_aug, reorder_state + ) + with torch.autograd.profiler.record_function( + "EnsembleModel: forward_decoder" + ): + lprobs, avg_attn_scores = self.model.forward_decoder( + tokens[:, : step + 1], + encoder_outs, + incremental_states, + self.temperature, + decoder_name=decoder_name, + encoder_outs_aug=encoder_outs_aug, + ) + + if self.lm_model is not None and not aux_task_name: + lm_out = self.lm_model(tokens[:, : step + 1]) + probs = self.lm_model.get_normalized_probs( + lm_out, log_probs=True, sample=None + ) + probs = probs[:, -1, :] * self.lm_weight + lprobs += probs + + lprobs[lprobs != lprobs] = torch.tensor(-math.inf).to(lprobs) + + lprobs[:, self.pad] = -math.inf # never select pad + lprobs[:, self.unk] -= self.unk_penalty # apply unk penalty + + # handle max length constraint + if step >= max_len: + lprobs[:, : self.eos] = -math.inf + lprobs[:, self.eos + 1 :] = -math.inf + + # handle prefix tokens (possibly with different lengths) + if ( + prefix_tokens is not None + and step < prefix_tokens.size(1) + and step < max_len + ): + lprobs, tokens, scores = self._prefix_tokens( + step, lprobs, scores, tokens, prefix_tokens, beam_size + ) + else: + if step < self.min_len: + # minimum length constraint (does not apply if using prefix_tokens) + lprobs[:, self.eos] = -math.inf + + if self.token_indices_to_suppress is not None: + lprobs[:, self.token_indices_to_suppress] = -math.inf + + # Record attention scores, only support avg_attn_scores is a Tensor + if avg_attn_scores is not None: + if attn is None: + attn = torch.empty( + bsz * beam_size, avg_attn_scores.size(1), max_len + 2 + ).to(scores) + attn[:, :, step + 1].copy_(avg_attn_scores) + + scores = scores.type_as(lprobs) + eos_bbsz_idx = torch.empty(0).to( + tokens + ) # indices of hypothesis ending with eos (finished sentences) + eos_scores = torch.empty(0).to( + scores + ) # scores of hypothesis ending with eos (finished sentences) + + if self.should_set_src_lengths: + self.search.set_src_lengths(src_lengths) + + if self.repeat_ngram_blocker is not None: + lprobs = self.repeat_ngram_blocker(tokens, lprobs, bsz, beam_size, step) + + # Shape: (batch, cand_size) + cand_scores, cand_indices, cand_beams = self.search.step( + step, + lprobs.view(bsz, -1, self.vocab_size), + scores.view(bsz, beam_size, -1)[:, :, :step], + tokens[:, : step + 1], + original_batch_idxs, + ) + + # cand_bbsz_idx contains beam indices for the top candidate + # hypotheses, with a range of values: [0, bsz*beam_size), + # and dimensions: [bsz, cand_size] + cand_bbsz_idx = cand_beams.add(bbsz_offsets) + + # finalize hypotheses that end in eos + # Shape of eos_mask: (batch size, beam size) + eos_mask = cand_indices.eq(self.eos) & cand_scores.ne(-math.inf) + eos_mask[:, :beam_size][cands_to_ignore] = torch.tensor(0).to(eos_mask) + + # only consider eos when it's among the top beam_size indices + # Now we know what beam item(s) to finish + # Shape: 1d list of absolute-numbered + eos_bbsz_idx = torch.masked_select( + cand_bbsz_idx[:, :beam_size], mask=eos_mask[:, :beam_size] + ) + + finalized_sents: List[int] = [] + if eos_bbsz_idx.numel() > 0: + eos_scores = torch.masked_select( + cand_scores[:, :beam_size], mask=eos_mask[:, :beam_size] + ) + + finalized_sents = self.finalize_hypos( + step, + eos_bbsz_idx, + eos_scores, + tokens, + scores, + finalized, + finished, + beam_size, + attn, + src_lengths, + max_len, + ) + num_remaining_sent -= len(finalized_sents) + + assert num_remaining_sent >= 0 + if num_remaining_sent == 0: + break + if self.search.stop_on_max_len and step >= max_len: + break + assert step < max_len, f"{step} < {max_len}" + + # Remove finalized sentences (ones for which {beam_size} + # finished hypotheses have been generated) from the batch. + if len(finalized_sents) > 0: + new_bsz = bsz - len(finalized_sents) + + # construct batch_idxs which holds indices of batches to keep for the next pass + batch_mask = torch.ones( + bsz, dtype=torch.bool, device=cand_indices.device + ) + batch_mask[finalized_sents] = False + # TODO replace `nonzero(as_tuple=False)` after TorchScript supports it + batch_idxs = torch.arange( + bsz, device=cand_indices.device + ).masked_select(batch_mask) + + # Choose the subset of the hypothesized constraints that will continue + self.search.prune_sentences(batch_idxs) + + eos_mask = eos_mask[batch_idxs] + cand_beams = cand_beams[batch_idxs] + bbsz_offsets.resize_(new_bsz, 1) + cand_bbsz_idx = cand_beams.add(bbsz_offsets) + cand_scores = cand_scores[batch_idxs] + cand_indices = cand_indices[batch_idxs] + + if prefix_tokens is not None: + prefix_tokens = prefix_tokens[batch_idxs] + src_lengths = src_lengths[batch_idxs] + cands_to_ignore = cands_to_ignore[batch_idxs] + + scores = scores.view(bsz, -1)[batch_idxs].view(new_bsz * beam_size, -1) + tokens = tokens.view(bsz, -1)[batch_idxs].view(new_bsz * beam_size, -1) + if attn is not None: + attn = attn.view(bsz, -1)[batch_idxs].view( + new_bsz * beam_size, attn.size(1), -1 + ) + bsz = new_bsz + else: + batch_idxs = None + + # Set active_mask so that values > cand_size indicate eos hypos + # and values < cand_size indicate candidate active hypos. + # After, the min values per row are the top candidate active hypos + + # Rewrite the operator since the element wise or is not supported in torchscript. + + eos_mask[:, :beam_size] = ~((~cands_to_ignore) & (~eos_mask[:, :beam_size])) + active_mask = torch.add( + eos_mask.type_as(cand_offsets) * cand_size, + cand_offsets[: eos_mask.size(1)], + ) + + # get the top beam_size active hypotheses, which are just + # the hypos with the smallest values in active_mask. + # {active_hypos} indicates which {beam_size} hypotheses + # from the list of {2 * beam_size} candidates were + # selected. Shapes: (batch size, beam size) + new_cands_to_ignore, active_hypos = torch.topk( + active_mask, k=beam_size, dim=1, largest=False + ) + + # update cands_to_ignore to ignore any finalized hypos. + cands_to_ignore = new_cands_to_ignore.ge(cand_size)[:, :beam_size] + # Make sure there is at least one active item for each sentence in the batch. + assert (~cands_to_ignore).any(dim=1).all() + + # update cands_to_ignore to ignore any finalized hypos + + # {active_bbsz_idx} denotes which beam number is continued for each new hypothesis (a beam + # can be selected more than once). + active_bbsz_idx = torch.gather(cand_bbsz_idx, dim=1, index=active_hypos) + active_scores = torch.gather(cand_scores, dim=1, index=active_hypos) + + active_bbsz_idx = active_bbsz_idx.view(-1) + active_scores = active_scores.view(-1) + + # copy tokens and scores for active hypotheses + + # Set the tokens for each beam (can select the same row more than once) + tokens[:, : step + 1] = torch.index_select( + tokens[:, : step + 1], dim=0, index=active_bbsz_idx + ) + # Select the next token for each of them + tokens.view(bsz, beam_size, -1)[:, :, step + 1] = torch.gather( + cand_indices, dim=1, index=active_hypos + ) + if step > 0: + scores[:, :step] = torch.index_select( + scores[:, :step], dim=0, index=active_bbsz_idx + ) + scores.view(bsz, beam_size, -1)[:, :, step] = torch.gather( + cand_scores, dim=1, index=active_hypos + ) + + # Update constraints based on which candidates were selected for the next beam + self.search.update_constraints(active_hypos) + + # copy attention for active hypotheses + if attn is not None: + attn[:, :, : step + 2] = torch.index_select( + attn[:, :, : step + 2], dim=0, index=active_bbsz_idx + ) + + # reorder incremental state in decoder + reorder_state = active_bbsz_idx + + # sort by score descending + for sent in range(len(finalized)): + scores = torch.tensor( + [float(elem["score"].item()) for elem in finalized[sent]] + ) + _, sorted_scores_indices = torch.sort(scores, descending=True) + finalized[sent] = [finalized[sent][ssi] for ssi in sorted_scores_indices] + finalized[sent] = torch.jit.annotate( + List[Dict[str, Tensor]], finalized[sent] + ) + return finalized + + +class EnsembleModel(EnsembleModelBase): + """A wrapper around an ensemble of models.""" + + def __init__(self, models): + super().__init__(models) + + @torch.jit.export + def forward_decoder( + self, + tokens, + encoder_outs: List[Dict[str, List[Tensor]]], + incremental_states: List[Dict[str, Dict[str, Optional[Tensor]]]], + temperature: float = 1.0, + decoder_name="decoder", + encoder_outs_aug: List[Dict[str, List[Tensor]]] = None, + ): + log_probs = [] + avg_attn: Optional[Tensor] = None + encoder_out: Optional[Dict[str, List[Tensor]]] = None + encoder_out_aug: Optional[Dict[str, List[Tensor]]] = None + for i, model in enumerate(self.models): + if self.has_encoder(): + encoder_out = encoder_outs[i] + if encoder_outs_aug is not None: + encoder_out_aug = encoder_outs_aug[i] + # decode each model + if self.has_incremental_states(): + if encoder_out_aug is not None: + decoder_out = getattr(model, decoder_name).forward( + tokens, + encoder_out=encoder_out, + encoder_out_aug=encoder_out_aug, + incremental_state=incremental_states[i], + ) + else: + decoder_out = getattr(model, decoder_name).forward( + tokens, + encoder_out=encoder_out, + incremental_state=incremental_states[i], + ) + else: + if hasattr(model, decoder_name): + decoder_out = getattr(model, decoder_name).forward( + tokens, encoder_out=encoder_out + ) + else: + decoder_out = model.forward(tokens) + + attn: Optional[Tensor] = None + decoder_len = len(decoder_out) + if decoder_len > 1 and decoder_out[1] is not None: + if isinstance(decoder_out[1], Tensor): + attn = decoder_out[1] + else: + attn_holder = decoder_out[1]["attn"] + if isinstance(attn_holder, Tensor): + attn = attn_holder + elif attn_holder is not None: + attn = attn_holder[0] + if attn is not None: + attn = attn[:, -1, :] + + decoder_out_tuple = ( + decoder_out[0][:, -1:, :].div_(temperature), + None if decoder_len <= 1 else decoder_out[1], + ) + probs = getattr(model, decoder_name).get_normalized_probs( + decoder_out_tuple, log_probs=True, sample=None + ) + probs = probs[:, -1, :] + if self.models_size == 1: + return probs, attn + + log_probs.append(probs) + if attn is not None: + if avg_attn is None: + avg_attn = attn + else: + avg_attn.add_(attn) + + avg_probs = torch.logsumexp(torch.stack(log_probs, dim=0), dim=0) - math.log( + self.models_size + ) + + if avg_attn is not None: + avg_attn.div_(self.models_size) + return avg_probs, avg_attn + + @torch.jit.export + def reorder_incremental_state( + self, + incremental_states: List[Dict[str, Dict[str, Optional[Tensor]]]], + new_order, + decoder_name="decoder", + ): + if not self.has_incremental_states(): + return + for i, model in enumerate(self.models): + getattr(model, decoder_name).reorder_incremental_state_scripting( + incremental_states[i], new_order + ) diff --git a/data/fairseq/examples/speech_to_speech/unity/sequence_generator_multi_decoder.py b/data/fairseq/examples/speech_to_speech/unity/sequence_generator_multi_decoder.py new file mode 100644 index 0000000000000000000000000000000000000000..af99a960b8b5c6d504f68613bc4e5f8e7af0a7aa --- /dev/null +++ b/data/fairseq/examples/speech_to_speech/unity/sequence_generator_multi_decoder.py @@ -0,0 +1,267 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +from typing import Dict, List, Optional + +import torch +import torch.nn as nn +from torch import Tensor + +from fairseq import search + + +class MultiDecoderSequenceGenerator(nn.Module): + def __init__( + self, + models, + tgt_dict, + tgt_dict_mt, + beam_size=1, + beam_size_mt=1, + max_len_a=0, + max_len_b=200, + max_len_a_mt=0, + max_len_b_mt=200, + max_len=0, + min_len=1, + normalize_scores=True, + len_penalty=1.0, + len_penalty_mt=1.0, + unk_penalty=0.0, + temperature=1.0, + match_source_len=False, + no_repeat_ngram_size=0, + eos=None, + eos_mt=None, + symbols_to_strip_from_output=None, + lm_model=None, + lm_weight=1.0, + ): + """Generates translations of a given source sentence. + + Args: + models (List[~fairseq.models.FairseqModel]): ensemble of models, + currently support fairseq.models.TransformerModel for scripting + beam_size (int, optional): beam width (default: 1) + max_len_a/b (int, optional): generate sequences of maximum length + ax + b, where x is the source length for the second pass + max_len_a_mt/b_mt (int, optional): generate sequences of maximum length + ax + b, where x is the source length for the first pass + max_len (int, optional): the maximum length of the generated output + (not including end-of-sentence) + min_len (int, optional): the minimum length of the generated output + (not including end-of-sentence) + normalize_scores (bool, optional): normalize scores by the length + of the output (default: True) + len_penalty (float, optional): length penalty in the second pass, where <1.0 favors + shorter, >1.0 favors longer sentences (default: 1.0) + len_penalty (float, optional): length penalty in the first pass, where <1.0 favors + shorter, >1.0 favors longer sentences (default: 1.0) + unk_penalty (float, optional): unknown word penalty, where <0 + produces more unks, >0 produces fewer (default: 0.0) + temperature (float, optional): temperature, where values + >1.0 produce more uniform samples and values <1.0 produce + sharper samples (default: 1.0) + match_source_len (bool, optional): outputs should match the source + length (default: False) + """ + super().__init__() + + from examples.speech_to_speech.unity.sequence_generator import SequenceGenerator + + self.generator = SequenceGenerator( + models, + tgt_dict, + beam_size=beam_size, + max_len_a=max_len_a, + max_len_b=max_len_b, + max_len=max_len, + min_len=min_len, + normalize_scores=normalize_scores, + len_penalty=len_penalty, + unk_penalty=unk_penalty, + temperature=temperature, + match_source_len=match_source_len, + no_repeat_ngram_size=no_repeat_ngram_size, + search_strategy=search.BeamSearch(tgt_dict), + eos=eos, + symbols_to_strip_from_output=symbols_to_strip_from_output, + lm_model=lm_model, + lm_weight=lm_weight, + ) + self.eos = self.generator.eos + + self.generator_mt = SequenceGenerator( + models, + tgt_dict_mt, + beam_size=beam_size_mt, + max_len_a=max_len_a_mt, + max_len_b=max_len_b_mt, + max_len=max_len, + min_len=min_len, + normalize_scores=normalize_scores, + len_penalty=len_penalty_mt, + unk_penalty=unk_penalty, + temperature=temperature, + match_source_len=match_source_len, + no_repeat_ngram_size=no_repeat_ngram_size, + search_strategy=search.BeamSearch(tgt_dict_mt), + eos=eos_mt, + symbols_to_strip_from_output=symbols_to_strip_from_output, + ) + + @torch.no_grad() + def generate( + self, models, sample: Dict[str, Dict[str, Tensor]], **kwargs + ) -> List[List[Dict[str, Tensor]]]: + """Generate translations. Match the api of other fairseq generators. + + Args: + models (List[~fairseq.models.FairseqModel]): ensemble of models + sample (dict): batch + prefix_tokens (torch.LongTensor, optional): force decoder to begin + with these tokens + constraints (torch.LongTensor, optional): force decoder to include + the list of constraints + bos_token (int, optional): beginning of sentence token + (default: self.eos) + """ + return self._generate(sample, **kwargs) + + def _generate( + self, + sample: Dict[str, Dict[str, Tensor]], + prefix_tokens: Optional[Tensor] = None, + constraints: Optional[Tensor] = None, + bos_token: Optional[int] = None, + ): + net_input = sample["net_input"] + + if "src_tokens" in net_input: + src_tokens = net_input["src_tokens"] + # length of the source text being the character length except EndOfSentence and pad + # if src_lengths exists in net_input (speech_to_text dataset case), then use it + if "src_lengths" in net_input: + src_lengths = net_input["src_lengths"] + else: + src_lengths = ( + ( + src_tokens.ne(self.generator.eos) + & src_tokens.ne(self.generator.pad) + ) + .long() + .sum(dim=1) + ) + else: + raise Exception( + "expected src_tokens or source in net input. input keys: " + + str(net_input.keys()) + ) + + if constraints is not None and not self.generator.search.supports_constraints: + raise NotImplementedError( + "Target-side constraints were provided, but search method doesn't support them" + ) + + # Initialize constraints, when active + self.generator.search.init_constraints(constraints, self.generator.beam_size) + self.generator_mt.search.init_constraints( + constraints, self.generator_mt.beam_size + ) + + # compute the encoder output for each beam + with torch.autograd.profiler.record_function("EnsembleModel: forward_encoder"): + encoder_outs = self.generator.model.forward_encoder(net_input) + + single_model = self.generator.model.single_model + mt_decoder = getattr(single_model, f"{single_model.mt_task_name}_decoder") + + # 1. MT decoder + finalized_mt = self.generator_mt.generate_decoder( + encoder_outs, + src_tokens, + src_lengths, + sample, + prefix_tokens, + constraints, + bos_token, + aux_task_name=single_model.mt_task_name, + ) + + # extract decoder output corresponding to the best hypothesis + max_tgt_len = max([len(hypo[0]["tokens"]) for hypo in finalized_mt]) + prev_output_tokens_mt = ( + src_tokens.new_zeros(src_tokens.shape[0], max_tgt_len) + .fill_(mt_decoder.padding_idx) + .int() + ) # B x T + for i, hypo in enumerate(finalized_mt): + i_beam = 0 + tmp = hypo[i_beam]["tokens"].int() # hyp + eos + prev_output_tokens_mt[i, 0] = self.generator_mt.eos + if tmp[-1] == self.generator_mt.eos: + tmp = tmp[:-1] + prev_output_tokens_mt[i, 1 : len(tmp) + 1] = tmp + + text = "".join([self.generator_mt.tgt_dict[c] for c in tmp]) + text = text.replace("_", " ") + text = text.replace("▁", " ") + text = text.replace("", " ") + text = text.replace("", "") + text = text.replace("", "") + if len(text) > 0 and text[0] == " ": + text = text[1:] + sample_id = sample["id"].tolist()[i] + print("{} (None-{})".format(text, sample_id)) + + x = mt_decoder( + prev_output_tokens_mt, + encoder_out=encoder_outs[0], + features_only=True, + )[0].transpose(0, 1) + + if getattr(single_model, "proj", None) is not None: + x = single_model.proj(x) + + mt_decoder_padding_mask = None + if prev_output_tokens_mt.eq(mt_decoder.padding_idx).any(): + mt_decoder_padding_mask = prev_output_tokens_mt.eq(mt_decoder.padding_idx) + + # 2. T2U encoder + if getattr(single_model, "synthesizer_encoder", None) is not None: + t2u_encoder_out = single_model.synthesizer_encoder( + x, + mt_decoder_padding_mask, + ) + else: + t2u_encoder_out = { + "encoder_out": [x], # T x B x C + "encoder_padding_mask": [mt_decoder_padding_mask] + if mt_decoder_padding_mask is not None + else [], # B x T + "encoder_embedding": [], + "encoder_states": [], + "src_tokens": [], + "src_lengths": [], + } + + if getattr(single_model, "t2u_augmented_cross_attn", False): + encoder_outs_aug = [t2u_encoder_out] + else: + encoder_outs = [t2u_encoder_out] + encoder_outs_aug = None + + # 3. T2U decoder + finalized = self.generator.generate_decoder( + encoder_outs, + src_tokens, + src_lengths, + sample, + prefix_tokens, + constraints, + bos_token, + encoder_outs_aug=encoder_outs_aug, + ) + return finalized diff --git a/data/fairseq/examples/textless_nlp/gslm/README.md b/data/fairseq/examples/textless_nlp/gslm/README.md new file mode 100644 index 0000000000000000000000000000000000000000..7a76ffd57c066c20af94aa3fca24c18e2ba4c3dd --- /dev/null +++ b/data/fairseq/examples/textless_nlp/gslm/README.md @@ -0,0 +1,21 @@ +# Generative Spoken Language Modeling + +* [Paper](https://arxiv.org/abs/2102.01192) +* [Demo](https://speechbot.github.io/gslm/index.html) + +We build and evaluate generative speech2speech systems using [Log Mel Filtebank](https://pytorch.org/audio/stable/compliance.kaldi.html#fbank), [Modified CPC](https://github.com/facebookresearch/CPC_audio), [HuBERT Base](https://github.com/pytorch/fairseq/tree/main/examples/hubert) and [Wav2Vec 2.0 Large](https://github.com/pytorch/fairseq/tree/main/examples/wav2vec). Our system is composed of three components, namely, *speech2unit*, *ulm* and *unit2speech*. We explain about models and usage of these components in their respective sub-directories. See the links below. + +## Speech to Unit Model (speech2unit) +Speech to unit model is used for quantizing raw speech into learned discrete speech units. [More details](speech2unit) + +## Unit Language Model (ulm) +Unit Language Model is a generative language model trained on discrete speech units. [More details](ulm) + +## Unit to Speech Model (unit2speech) +Unit to speech model is used for synthesizing speech from discrete speech units. [More details](unit2speech) + +## Metrics +We show how to compute ASR based metrics as well as zero-shot metrics proposed in our paper [here](metrics). + +## Tools +We share two tools to resynthesize a given spoken utterance, and generate novel spoken language given a spoken prompt. [More detail](tools) diff --git a/data/fairseq/examples/textless_nlp/gslm/unit2speech/convert_to_16k.py b/data/fairseq/examples/textless_nlp/gslm/unit2speech/convert_to_16k.py new file mode 100644 index 0000000000000000000000000000000000000000..2be848fceae65e3bd5747a2c98106b0215c6a039 --- /dev/null +++ b/data/fairseq/examples/textless_nlp/gslm/unit2speech/convert_to_16k.py @@ -0,0 +1,56 @@ +import os +import shlex +import subprocess +import progressbar +from time import time +from pathlib import Path + +def find_all_files(path_dir, extension): + out = [] + for root, dirs, filenames in os.walk(path_dir): + for f in filenames: + if f.endswith(extension): + out.append(((str(Path(f).stem)), os.path.join(root, f))) + return out + +def convert16k(inputfile, outputfile16k): + command = ('sox -c 1 -b 16 {} -t wav {} rate 16k'.format(inputfile, outputfile16k)) + subprocess.call(shlex.split(command)) + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser(description='Convert to wav 16k audio using sox.') + parser.add_argument('input_dir', type=str, + help='Path to the input dir.') + parser.add_argument('output_dir', type=str, + help='Path to the output dir.') + parser.add_argument('--extension', type=str, default='wav', + help='Audio file extension in the input. Default: mp3') + args = parser.parse_args() + + # Find all sequences + print(f"Finding all audio files with extension '{args.extension}' from {args.input_dir}...") + audio_files = find_all_files(args.input_dir, args.extension) + print(f"Done! Found {len(audio_files)} files.") + + # Convert to relative path + audio_files = [os.path.relpath(file[-1], start=args.input_dir) for file in audio_files] + + # Create all the directories needed + rel_dirs_set = set([os.path.dirname(file) for file in audio_files]) + for rel_dir in rel_dirs_set: + Path(os.path.join(args.output_dir, rel_dir)).mkdir(parents=True, exist_ok=True) + + # Converting wavs files + print("Converting the audio to wav files...") + bar = progressbar.ProgressBar(maxval=len(audio_files)) + bar.start() + start_time = time() + for index, file in enumerate(audio_files): + bar.update(index) + input_file = os.path.join(args.input_dir, file) + output_file = os.path.join(args.output_dir, os.path.splitext(file)[0]+".wav") + convert16k(input_file, output_file) + bar.finish() + print(f"...done {len(audio_files)} files in {time()-start_time} seconds.") \ No newline at end of file diff --git a/data/fairseq/examples/textless_nlp/gslm/unit2speech/glow.py b/data/fairseq/examples/textless_nlp/gslm/unit2speech/glow.py new file mode 100644 index 0000000000000000000000000000000000000000..41fd437feb17a5b40d4cbc43a0ab4e69867618bc --- /dev/null +++ b/data/fairseq/examples/textless_nlp/gslm/unit2speech/glow.py @@ -0,0 +1,312 @@ +# ***************************************************************************** +# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of the NVIDIA CORPORATION nor the +# names of its contributors may be used to endorse or promote products +# derived from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +# DISCLAIMED. IN NO EVENT SHALL NVIDIA CORPORATION BE LIABLE FOR ANY +# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +# +# ***************************************************************************** +import copy +import torch +from torch.autograd import Variable +import torch.nn.functional as F + + +@torch.jit.script +def fused_add_tanh_sigmoid_multiply(input_a, input_b, n_channels): + n_channels_int = n_channels[0] + in_act = input_a+input_b + t_act = torch.tanh(in_act[:, :n_channels_int, :]) + s_act = torch.sigmoid(in_act[:, n_channels_int:, :]) + acts = t_act * s_act + return acts + + +class WaveGlowLoss(torch.nn.Module): + def __init__(self, sigma=1.0): + super(WaveGlowLoss, self).__init__() + self.sigma = sigma + + def forward(self, model_output): + z, log_s_list, log_det_W_list = model_output + for i, log_s in enumerate(log_s_list): + if i == 0: + log_s_total = torch.sum(log_s) + log_det_W_total = log_det_W_list[i] + else: + log_s_total = log_s_total + torch.sum(log_s) + log_det_W_total += log_det_W_list[i] + + loss = torch.sum(z*z)/(2*self.sigma*self.sigma) - log_s_total - log_det_W_total + return loss/(z.size(0)*z.size(1)*z.size(2)) + + +class Invertible1x1Conv(torch.nn.Module): + """ + The layer outputs both the convolution, and the log determinant + of its weight matrix. If reverse=True it does convolution with + inverse + """ + def __init__(self, c): + super(Invertible1x1Conv, self).__init__() + self.conv = torch.nn.Conv1d(c, c, kernel_size=1, stride=1, padding=0, + bias=False) + + # Sample a random orthonormal matrix to initialize weights + _qr = torch.linalg.qr if torch.__version__ >= "1.8" else torch.qr + W = _qr(torch.FloatTensor(c, c).normal_())[0] + + # Ensure determinant is 1.0 not -1.0 + if torch.det(W) < 0: + W[:,0] = -1*W[:,0] + W = W.view(c, c, 1) + self.conv.weight.data = W + + def forward(self, z, reverse=False): + # shape + batch_size, group_size, n_of_groups = z.size() + + W = self.conv.weight.squeeze() + + if reverse: + if not hasattr(self, 'W_inverse'): + # Reverse computation + W_inverse = W.float().inverse() + W_inverse = Variable(W_inverse[..., None]) + if z.type() == 'torch.cuda.HalfTensor': + W_inverse = W_inverse.half() + self.W_inverse = W_inverse + z = F.conv1d(z, self.W_inverse, bias=None, stride=1, padding=0) + return z + else: + # Forward computation + log_det_W = batch_size * n_of_groups * torch.logdet(W) + z = self.conv(z) + return z, log_det_W + + +class WN(torch.nn.Module): + """ + This is the WaveNet like layer for the affine coupling. The primary difference + from WaveNet is the convolutions need not be causal. There is also no dilation + size reset. The dilation only doubles on each layer + """ + def __init__(self, n_in_channels, n_mel_channels, n_layers, n_channels, + kernel_size): + super(WN, self).__init__() + assert(kernel_size % 2 == 1) + assert(n_channels % 2 == 0) + self.n_layers = n_layers + self.n_channels = n_channels + self.in_layers = torch.nn.ModuleList() + self.res_skip_layers = torch.nn.ModuleList() + + start = torch.nn.Conv1d(n_in_channels, n_channels, 1) + start = torch.nn.utils.weight_norm(start, name='weight') + self.start = start + + # Initializing last layer to 0 makes the affine coupling layers + # do nothing at first. This helps with training stability + end = torch.nn.Conv1d(n_channels, 2*n_in_channels, 1) + end.weight.data.zero_() + end.bias.data.zero_() + self.end = end + + cond_layer = torch.nn.Conv1d(n_mel_channels, 2*n_channels*n_layers, 1) + self.cond_layer = torch.nn.utils.weight_norm(cond_layer, name='weight') + + for i in range(n_layers): + dilation = 2 ** i + padding = int((kernel_size*dilation - dilation)/2) + in_layer = torch.nn.Conv1d(n_channels, 2*n_channels, kernel_size, + dilation=dilation, padding=padding) + in_layer = torch.nn.utils.weight_norm(in_layer, name='weight') + self.in_layers.append(in_layer) + + + # last one is not necessary + if i < n_layers - 1: + res_skip_channels = 2*n_channels + else: + res_skip_channels = n_channels + res_skip_layer = torch.nn.Conv1d(n_channels, res_skip_channels, 1) + res_skip_layer = torch.nn.utils.weight_norm(res_skip_layer, name='weight') + self.res_skip_layers.append(res_skip_layer) + + def forward(self, forward_input): + audio, spect = forward_input + audio = self.start(audio) + output = torch.zeros_like(audio) + n_channels_tensor = torch.IntTensor([self.n_channels]) + + spect = self.cond_layer(spect) + + for i in range(self.n_layers): + spect_offset = i*2*self.n_channels + acts = fused_add_tanh_sigmoid_multiply( + self.in_layers[i](audio), + spect[:,spect_offset:spect_offset+2*self.n_channels,:], + n_channels_tensor) + + res_skip_acts = self.res_skip_layers[i](acts) + if i < self.n_layers - 1: + audio = audio + res_skip_acts[:,:self.n_channels,:] + output = output + res_skip_acts[:,self.n_channels:,:] + else: + output = output + res_skip_acts + + return self.end(output) + + +class WaveGlow(torch.nn.Module): + def __init__(self, n_mel_channels, n_flows, n_group, n_early_every, + n_early_size, WN_config): + super(WaveGlow, self).__init__() + + self.upsample = torch.nn.ConvTranspose1d(n_mel_channels, + n_mel_channels, + 1024, stride=256) + assert(n_group % 2 == 0) + self.n_flows = n_flows + self.n_group = n_group + self.n_early_every = n_early_every + self.n_early_size = n_early_size + self.WN = torch.nn.ModuleList() + self.convinv = torch.nn.ModuleList() + + n_half = int(n_group/2) + + # Set up layers with the right sizes based on how many dimensions + # have been output already + n_remaining_channels = n_group + for k in range(n_flows): + if k % self.n_early_every == 0 and k > 0: + n_half = n_half - int(self.n_early_size/2) + n_remaining_channels = n_remaining_channels - self.n_early_size + self.convinv.append(Invertible1x1Conv(n_remaining_channels)) + self.WN.append(WN(n_half, n_mel_channels*n_group, **WN_config)) + self.n_remaining_channels = n_remaining_channels # Useful during inference + + def forward(self, forward_input): + """ + forward_input[0] = mel_spectrogram: batch x n_mel_channels x frames + forward_input[1] = audio: batch x time + """ + spect, audio = forward_input + + # Upsample spectrogram to size of audio + spect = self.upsample(spect) + assert(spect.size(2) >= audio.size(1)) + if spect.size(2) > audio.size(1): + spect = spect[:, :, :audio.size(1)] + + spect = spect.unfold(2, self.n_group, self.n_group).permute(0, 2, 1, 3) + spect = spect.contiguous().view(spect.size(0), spect.size(1), -1).permute(0, 2, 1) + + audio = audio.unfold(1, self.n_group, self.n_group).permute(0, 2, 1) + output_audio = [] + log_s_list = [] + log_det_W_list = [] + + for k in range(self.n_flows): + if k % self.n_early_every == 0 and k > 0: + output_audio.append(audio[:,:self.n_early_size,:]) + audio = audio[:,self.n_early_size:,:] + + audio, log_det_W = self.convinv[k](audio) + log_det_W_list.append(log_det_W) + + n_half = int(audio.size(1)/2) + audio_0 = audio[:,:n_half,:] + audio_1 = audio[:,n_half:,:] + + output = self.WN[k]((audio_0, spect)) + log_s = output[:, n_half:, :] + b = output[:, :n_half, :] + audio_1 = torch.exp(log_s)*audio_1 + b + log_s_list.append(log_s) + + audio = torch.cat([audio_0, audio_1],1) + + output_audio.append(audio) + return torch.cat(output_audio,1), log_s_list, log_det_W_list + + def infer(self, spect, sigma=1.0): + spect = self.upsample(spect) + # trim conv artifacts. maybe pad spec to kernel multiple + time_cutoff = self.upsample.kernel_size[0] - self.upsample.stride[0] + spect = spect[:, :, :-time_cutoff] + + spect = spect.unfold(2, self.n_group, self.n_group).permute(0, 2, 1, 3) + spect = spect.contiguous().view(spect.size(0), spect.size(1), -1).permute(0, 2, 1) + + if spect.type() == 'torch.cuda.HalfTensor': + audio = torch.cuda.HalfTensor(spect.size(0), + self.n_remaining_channels, + spect.size(2)).normal_() + else: + audio = torch.cuda.FloatTensor(spect.size(0), + self.n_remaining_channels, + spect.size(2)).normal_() + + audio = torch.autograd.Variable(sigma*audio) + + for k in reversed(range(self.n_flows)): + n_half = int(audio.size(1)/2) + audio_0 = audio[:,:n_half,:] + audio_1 = audio[:,n_half:,:] + + output = self.WN[k]((audio_0, spect)) + + s = output[:, n_half:, :] + b = output[:, :n_half, :] + audio_1 = (audio_1 - b)/torch.exp(s) + audio = torch.cat([audio_0, audio_1],1) + + audio = self.convinv[k](audio, reverse=True) + + if k % self.n_early_every == 0 and k > 0: + if spect.type() == 'torch.cuda.HalfTensor': + z = torch.cuda.HalfTensor(spect.size(0), self.n_early_size, spect.size(2)).normal_() + else: + z = torch.cuda.FloatTensor(spect.size(0), self.n_early_size, spect.size(2)).normal_() + audio = torch.cat((sigma*z, audio),1) + + audio = audio.permute(0,2,1).contiguous().view(audio.size(0), -1).data + return audio + + @staticmethod + def remove_weightnorm(model): + waveglow = model + for WN in waveglow.WN: + WN.start = torch.nn.utils.remove_weight_norm(WN.start) + WN.in_layers = remove(WN.in_layers) + WN.cond_layer = torch.nn.utils.remove_weight_norm(WN.cond_layer) + WN.res_skip_layers = remove(WN.res_skip_layers) + return waveglow + + +def remove(conv_list): + new_conv_list = torch.nn.ModuleList() + for old_conv in conv_list: + old_conv = torch.nn.utils.remove_weight_norm(old_conv) + new_conv_list.append(old_conv) + return new_conv_list diff --git a/data/fairseq/examples/textless_nlp/gslm/unit2speech/multiproc.py b/data/fairseq/examples/textless_nlp/gslm/unit2speech/multiproc.py new file mode 100644 index 0000000000000000000000000000000000000000..2a287a4e97c66acbd36897b25f2ece5494005f03 --- /dev/null +++ b/data/fairseq/examples/textless_nlp/gslm/unit2speech/multiproc.py @@ -0,0 +1,27 @@ +import os +import time +import torch +import sys +import subprocess + +argslist = list(sys.argv)[1:] +log_dir = argslist[-1] +num_gpus = torch.cuda.device_count() +argslist.append('--n_gpus={}'.format(num_gpus)) +workers = [] +job_id = time.strftime("%Y_%m_%d-%H%M%S") +argslist.append("--group_name=group_{}".format(job_id)) + +print("GPU log directory is {}".format(log_dir)) +os.makedirs(log_dir, exist_ok=True) +for i in range(num_gpus): + argslist.append('--rank={}'.format(i)) + stdout = None if i == 0 else open("{}/{}_GPU_{}.log".format(log_dir, job_id, i), + "w") + print(argslist) + p = subprocess.Popen([str(sys.executable)]+argslist, stdout=stdout) + workers.append(p) + argslist = argslist[:-1] + +for p in workers: + p.wait() diff --git a/data/fairseq/examples/textless_nlp/gslm/unit2speech/synthesize_audio_from_units.py b/data/fairseq/examples/textless_nlp/gslm/unit2speech/synthesize_audio_from_units.py new file mode 100644 index 0000000000000000000000000000000000000000..80730843bf6ae5bbb10e190df38941f39b14cb9a --- /dev/null +++ b/data/fairseq/examples/textless_nlp/gslm/unit2speech/synthesize_audio_from_units.py @@ -0,0 +1,105 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import argparse +import logging +import os + +import soundfile as sf +from examples.textless_nlp.gslm.unit2speech.tts_data import ( + TacotronInputDataset, +) +from examples.textless_nlp.gslm.unit2speech.utils import ( + load_quantized_audio_from_file, + load_tacotron, + load_waveglow, + synthesize_audio, +) + + +def get_logger(): + log_format = "[%(asctime)s] [%(levelname)s]: %(message)s" + logging.basicConfig(format=log_format, level=logging.INFO) + logger = logging.getLogger(__name__) + return logger + + +def get_parser(): + parser = argparse.ArgumentParser( + description="Wav2Vec 2.0 speech generator." + ) + parser.add_argument( + "--quantized_unit_path", + type=str, + help="K-means model file path to use for inference", + ) + parser.add_argument( + "--tts_model_path", + type=str, + help="TTS model file path to use for inference", + ) + parser.add_argument( + "--waveglow_path", + type=str, + help="Path to the waveglow checkpoint (vocoder).", + ) + parser.add_argument( + "--code_dict_path", + type=str, + help="Code dict file path to use for inference", + ) + parser.add_argument("--max_decoder_steps", type=int, default=2000) + parser.add_argument("--denoiser_strength", type=float, default=0.1) + parser.add_argument( + "--out_audio_dir", + type=str, + help="Output directory to dump audio files", + ) + + return parser + + +def main(args, logger): + # Load quantized audio + logger.info(f"Loading quantized audio from {args.quantized_unit_path}...") + names_batch, quantized_units_batch = load_quantized_audio_from_file( + file_path=args.quantized_unit_path + ) + + logger.info(f"Loading TTS model from {args.tts_model_path}...") + tacotron_model, sample_rate, hparams = load_tacotron( + tacotron_model_path=args.tts_model_path, + max_decoder_steps=args.max_decoder_steps, + ) + + logger.info(f"Loading Waveglow model from {args.waveglow_path}...") + waveglow, denoiser = load_waveglow(waveglow_path=args.waveglow_path) + + if not os.path.exists(hparams.code_dict): + hparams.code_dict = args.code_dict_path + tts_dataset = TacotronInputDataset(hparams) + + for name, quantized_units in zip(names_batch, quantized_units_batch): + quantized_units_str = " ".join(map(str, quantized_units)) + tts_input = tts_dataset.get_tensor(quantized_units_str) + mel, aud, aud_dn, has_eos = synthesize_audio( + tacotron_model, + waveglow, + denoiser, + tts_input.unsqueeze(0), + strength=args.denoiser_strength, + ) + out_file_path = os.path.join(args.out_audio_dir, f"{name}.wav") + sf.write( + f"{out_file_path}", aud_dn[0].cpu().float().numpy(), sample_rate + ) + + +if __name__ == "__main__": + parser = get_parser() + args = parser.parse_args() + logger = get_logger() + logger.info(args) + main(args, logger) diff --git a/data/fairseq/examples/textless_nlp/gslm/unit2speech/tts_data.py b/data/fairseq/examples/textless_nlp/gslm/unit2speech/tts_data.py new file mode 100644 index 0000000000000000000000000000000000000000..d2b04c0feea0f7656f426e63090a6d4fb5fb8ed8 --- /dev/null +++ b/data/fairseq/examples/textless_nlp/gslm/unit2speech/tts_data.py @@ -0,0 +1,54 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + + +import torch +import numpy as np +from examples.textless_nlp.gslm.unit2speech.tacotron2.text import ( + EOS_TOK, + SOS_TOK, + code_to_sequence, + text_to_sequence, +) +from examples.textless_nlp.gslm.unit2speech.tacotron2.utils import ( + load_code_dict, +) + + +class TacotronInputDataset: + def __init__(self, hparams, append_str=""): + self.is_text = getattr(hparams, "text_or_code", "text") == "text" + if not self.is_text: + self.code_dict = load_code_dict( + hparams.code_dict, hparams.add_sos, hparams.add_eos + ) + self.code_key = hparams.code_key + self.add_sos = hparams.add_sos + self.add_eos = hparams.add_eos + self.collapse_code = hparams.collapse_code + self.append_str = append_str + + def process_code(self, inp_str): + inp_toks = inp_str.split() + if self.add_sos: + inp_toks = [SOS_TOK] + inp_toks + if self.add_eos: + inp_toks = inp_toks + [EOS_TOK] + return code_to_sequence(inp_toks, self.code_dict, self.collapse_code) + + def process_text(self, inp_str): + return text_to_sequence(inp_str, ["english_cleaners"]) + + def get_tensor(self, inp_str): + # uid, txt, inp_str = self._get_data(idx) + inp_str = inp_str + self.append_str + if self.is_text: + inp_toks = self.process_text(inp_str) + else: + inp_toks = self.process_code(inp_str) + return torch.from_numpy(np.array(inp_toks)).long() + + def __len__(self): + return len(self.data) diff --git a/data/fairseq/examples/textless_nlp/pgslm/README.md b/data/fairseq/examples/textless_nlp/pgslm/README.md new file mode 100644 index 0000000000000000000000000000000000000000..596467fe8283a7f1514261151f728f7bd233dbd7 --- /dev/null +++ b/data/fairseq/examples/textless_nlp/pgslm/README.md @@ -0,0 +1,318 @@ +# Text-Free Prosody-Aware Generative Spoken Language Modeling + +This folder contains code and recipes to reproduce results reported in a paper _Text-Free Prosody-Aware Generative Spoken Language Modeling_, +Eugene Kharitonov*, Ann Lee*, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu-Anh Nguyen, Morgane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, Wei-Ning Hsu, 2021. arxiv/2109.03264 [[arxiv]](https://arxiv.org/abs/2109.03264). + +`*` denotes equal contribution. + +You can find demo samples [[here]](https://speechbot.github.io/pgslm/index.html). + +
+ If you find this code useful, please consider citing our work using this bibtex + +``` + @misc{Kharitonov2021, + title={Text-Free Prosody-Aware Generative Spoken Language Modeling}, + author={Eugene Kharitonov and Ann Lee and Adam Polyak and Yossi Adi and Jade Copet and Kushal Lakhotia and Tu-Anh Nguyen and Morgane Rivière and Abdelrahman Mohamed and Emmanuel Dupoux and Wei-Ning Hsu}, + year={2021}, + eprint={2109.03264}, + archivePrefix={arXiv}, + primaryClass={cs.CL} +} +``` +
+ + +## Additional requirements +Three packages are required in addition to fairseq, they are installable with pip: +```bash +pip install AMFM-decompy SoundFile scipy sklearn torchaudio npy-append-array +``` + +## Data preprocessing + +### Prepare unit pseudo-text transcriptions of the audio +To get unit trascripts of the speech data we rely on the preprocessing steps of [GSLM](https://github.com/pytorch/fairseq/tree/main/examples/textless_nlp/gslm/speech2unit/) work. + +Firstly, we will need to prepare manifest files for the dataset we want to preprocess +``` +mkdir manifests/ +python examples/wav2vec/wav2vec_manifest.py --valid-percent=0.0 $DATA_PATH --dest=manifests/train/ +``` +Next, we need a pre-trained HuBERT-base-ls960 model [[download]](https://dl.fbaipublicfiles.com/hubert/hubert_base_ls960.pt) and a corresponding kmeans-100 quantizer [[download]](https://dl.fbaipublicfiles.com/textless_nlp/gslm/hubert/km100/km.bin). Having those we can quantize the dataset: +``` +python examples/textless_nlp/gslm/speech2unit/clustering/quantize_with_kmeans.py \ + --feature_type hubert \ + --kmeans_model_path km.bin \ + --acoustic_model_path hubert_base_ls960.pt \ + --layer 6 \ + --manifest_path manifests/train/train.tsv \ + --out_quantized_file_path manifests/train/units +``` + +Finally, by running +``` +python examples/textless_nlp/pgslm/scripts/join_units_manifest.py --manifest=manifests/train/train.tsv --units=manifests/train/units --output=train.txt +``` +We will get the training data description `train.txt` in the format that pGSLM expects. The above steps have to be repeated for +dev/test sets. Importantly, we rely on an assumption that the directories are structured as in LibriSpeech, i.e. the file paths follow the +`//.wav` format. + +### Preprocess data for pGSLM +The very first step is to obtain the F0 quantization bins. +Assume the vocoder training manifest is `vocoder_train.txt` (in pGSLM data format prepared with the same process above). +We prepare the quantized F0 from the vocoder training data by running +```sh +bash examples/textless_nlp/pgslm/scripts/prepare_f0_quantization.sh \ + vocoder_train.txt 32 # we use 32 bins in the paper +``` +- ``: sampling rate of the audio files in the manifest +- ``: where to output the output files +- ``: prefix of the output files + +The script will generate +- `.f0_stat.pt`: the speaker-level F0 statistics, which can be used in vocoder training +- `_mean_norm_log_f0_bin.th`: the quantized F0, which should be used in `prepare_data.sh` below + +**Note:** See "Pre-trained models" for the pre-computed speaker-level F0 statistics and quantized F0 bins. We suggest using the pre-computed statistics for the data preparation below in order to take advantage of the pre-trained vocoder for waveform generation. + +Next prepare the pGSLM data. +Assume train/valid/test manifests are `{train,valid,test}.txt`. +Here is an example of how to preprocess data: + +```sh +bash examples/textless_nlp/pgslm/scripts/prepare_data.sh \ + train.txt valid.txt test.txt \ + /_mean_norm_log_f0_bin.th +``` +- ``: discrete unit vocabulary size (we used a kmeans quantizer with the number of units equal to 100 in the example above) +- ``: downsampling rate relative to the waveform (e.g., 320 for HuBERT units) +- ``: sampling rate of the audio files in the manifest +- ``: where to output the preprocessed files + +This will create the dataset json config used for the next section at +`/data_config.json`. + +Note that the example script uses only one thread to compute F0, which can take +_very long_ for preprocessing large datasets. It is suggested to distribute +jobs over multiple nodes/processes with `--nshards=x` and `--rank=z` (where z is +in [1, x]) in `preprocess_f0.py`, and set `--nshards_list=x` in +`prepare_data.py` correspondingly to collect sharded F0 data. + +Now, everything is ready for training a model. + +## Training Multi-Stream Transformer Unit Language Model (MS-TLM) + +Below is an example command that trains Multi-Stream Transformer Language Model (MS-TLM) on a prepared dataset: +```bash +DATASET=data_config.json + +fairseq-train $DATASET \ + --task=speech_unit_modeling \ + --arch="transformer_ulm_tiny" \ + --criterion=speech_unit_lm_criterion \ + --share-decoder-input-output-embed \ + --dropout=0.1 \ + --attention-dropout=0.1 \ + --optimizer="adam" \ + --adam-betas="(0.9, 0.98)" \ + --clip-norm=1.0 \ + --lr=0.0005 \ + --lr-scheduler="inverse_sqrt" \ + --warmup-updates=4000 \ + --warmup-init-lr=1e-07 \ + --tokens-per-sample=3072 \ + --max-tokens=3072 \ + --update-freq=4 \ + --max-epoch=70 \ + --num-workers=0 \ + --skip-invalid-size-inputs-valid-test \ + --loss-weights="1.0;0.5;0.0" \ + --ignore-f0-input \ + --checkpoint-activations \ + --fp16 \ + --max-target-positions=4096 \ + --stream-shifts="1,1" \ + --log-f0 --normalize-f0-mean --interpolate-f0 \ + --ignore-unused-valid-subsets \ + --discrete-duration --discrete-f0 +``` + +Some of the important parameters that are specific to MS-TLM: + * `arch`: specifies the Transformer architecture used. Supported options are: + * `transformer_ulm_tiny` - a tiny model that can be used for debugging; it has 2 layers, 1 attention head, FFN and embedding dimensions of 64, + * `transformer_ulm` - a base model with 6 layers, 8 heads, embedding dimension 512, and FFN dimensionality of 2048, + * `transformer_ulm_big` - the largest model we experiment with in the paper: 12-layer/16 heads, 1024/4096 embedding and FFN dimensions; + * `loss-weights`: this parameter sets importance weights (must be non-negative) for the components of the loss that correspond to unit, duration, and F0 streams. To turn off a component of the loss, its weight has to be set to 0. For instance, to predict only unit stream the parameter should be set to "1;0;0"; + * `stream-shifts`: specifies relative shifts of the two prosodic streams w.r.t. the unit stream (duration and F0, respectively). No shift corresponds to "0,0"; + * `ignore-duration-input`/`ignore-f0-input`: setting these flags would zero-out correpsonding input streams; + * `max-token-duration`: duration values would be max-capped by the specified value; + * `discrete-duration`/`discrete-f0`: whether duration and F0 streams should be quantized; + * `log_f0`, `normalize-f0-mean`, `normalize-f0-std`, `interpolate-f0`: configure how F0 stream is treated. `log_f0` sets up modelling in the log-space, `normalize-f0-mean`/`normalize-f0-std` control per-speaker normalization, and `interpolate-f0` enables F0 interpolation for unvoiced regions where F0 was set to 0, + * `mask-dur-prob`, `mask-f0-prob`, `mask-dur-seg-prob`, `mask-f0-seg-prob`, `mask-unit-seg-prob`, `mask-unit-seg-leng`: this family of parameters sets the probababilities of masking individual steps and spans on each stream as well as lengths of the maked spans. + + +## Pre-trained models +### MS-TLM +Below you can find checkpoints for four best-performing models from the paper (IDs 9..12 in Table 1). These models are trained on Hubert-100 transcripts of the LibriLight-6K dataset. They have the prosody streams shifted by 1 w.r.t. the unit stream. All models predict all three streams (units, duration, and F0), but two +of them only have unit steam in their input. + +| | Continuous prosody | Quantized prosody | +|-------------------|--------------------|-------------------| +| No prosody input | [[download]](https://dl.fbaipublicfiles.com/textless_nlp/pgslm/ulm_checkpoints/continuous_no_prosody_shift_1_1.pt) | [[download]](https://dl.fbaipublicfiles.com/textless_nlp/pgslm/ulm_checkpoints/discrete_no_prosody_shift_1_1.pt) | +| Has prosody input | [[download]](https://dl.fbaipublicfiles.com/textless_nlp/pgslm/ulm_checkpoints/continuous_prosody_shift_1_1.pt) | [[download]](https://dl.fbaipublicfiles.com/textless_nlp/pgslm/ulm_checkpoints/discrete_prosody_shift_1_1.pt)| + +The optimal per-stream sampling temperatures/scaling parameters that we have identified for these models, in the (`T-token, T-duration, T-f0`) format: + +| | Continuous prosody | Quantized prosody | +|-------------------|--------------------|-------------------| +| No prosody input | 0.7, 0.125, 0.0003125| 0.7, 0.25, 0.5 | +| Has prosody input | 0.7, 0.125, 0.00125 | 0.7, 0.25, 0.7 | + +## Vocoder +| Units | Prosody | F0 stats | Checkpoint | Config | +|-------------------|---------|--------------|------------|--------| +| HuBERT-base-ls960, kmeans-100 | [[Quantized 32 bins]](https://dl.fbaipublicfiles.com/textless_nlp/pgslm/vocoder/blizzard2013/mean_norm_log_f0_seg_bin.th) | [[download]](https://dl.fbaipublicfiles.com/textless_nlp/pgslm/vocoder/blizzard2013/f0_stats.pt) | [[download]](https://dl.fbaipublicfiles.com/textless_nlp/pgslm/vocoder/blizzard2013/naive_quant_32_norm_log_seg_hubert/checkpoint.pt) | [[download]](https://dl.fbaipublicfiles.com/textless_nlp/pgslm/vocoder/blizzard2013/naive_quant_32_norm_log_seg_hubert/config.json) | +| HuBERT-base-ls960, kmeans-100 | Continuous | [[download]](https://dl.fbaipublicfiles.com/textless_nlp/pgslm/vocoder/blizzard2013/f0_stats.pt) | [[download]](https://dl.fbaipublicfiles.com/textless_nlp/pgslm/vocoder/blizzard2013/mean_norm_log_f0_hubert/checkpoint.pt) | [[download]](https://dl.fbaipublicfiles.com/textless_nlp/pgslm/vocoder/blizzard2013/mean_norm_log_f0_hubert/config.json) | + + +## Evaluating a trained model +Evaluation is done with the `eval/cont_metrics.py` scripts. As described in the paper, there are several metrics used. + +**Teacher-forced metrics** +```bash +SET=valid +CHECKPOINT_PATH=discrete_prosody_shift_1_1.pt +DATA=data_config.json + +python examples/textless_nlp/pgslm/eval/cont_metrics.py $DATA \ + --metric=teacher_force_everything \ + --path=$CHECKPOINT_PATH \ + --batch-size=16 \ + --fp16 \ + --seed=111 \ + --eval-subset=$SET \ + --f0-discretization-bounds=mean_norm_log_f0_seg_bin.th --dequantize-prosody +``` +(Using this command, our provided `discrete_prosody_shift_1_1.pt` checkpoint should produce `{'token_loss': 1.408..., 'duration_loss': 0.5424..., 'f0_loss': 0.0474...}` on LibriSpeech dev-clean). + +The parameters `--f0-discretization-bounds=mean_norm_log_f0_seg_bin.th --dequantize-prosody` are specific for quantized-prosody models. They signal that the prosody streams must be decoded into the continuous domain before calculating correlation. It is the same `*_mean_norm_log_f0_bin.th` file as we prepared before. +The `mean_norm_log_f0_seg_bin.th` file we used with the pre-trained models can be downloaded [[here]](https://dl.fbaipublicfiles.com/textless_nlp/pgslm/vocoder/blizzard2013/mean_norm_log_f0_seg_bin.th). + + +**Consistency (aka Correlation) metrics** + +The following command estimates correlation between mean values of the F0 stream in the prompt and in the generated continuation (unit and duration steams are fixed). + +```bash +T_F0=0.7 +EXPLOSION=20 +SET=test +CHECKPOINT_PATH=discrete_prosody_shift_1_1.pt +DATA=data_config.json + +python examples/textless_nlp/pgslm/eval/cont_metrics.py $DATA \ + --prefix-length=150 \ + --metric=correlation \ + --path=$CHECKPOINT_PATH \ + --batch-size=16 \ + --fp16 \ + --seed=111 \ + --teacher-force-tokens \ + --teacher-force-duration \ + --min-length=300 \ + --batch-explosion-rate=$EXPLOSION \ + --T-f0=$T_F0 \ + --eval-subset=$SET \ + --f0-discretization-bounds=mean_norm_log_f0_seg_bin.th \ + --dequantize-prosody --n-workers=8 +``` +(Using this command, our provided `discrete_prosody_shift_1_1.pt` checkpoint should produce `{...'F0 corr': 0.315 ..}` on LibriSpeech test-clean). + + * By using flags `--teacher-force-tokens, --teacher-force-duration, --teacher-force-f0` one can calculate correlations along each stream while having other two streams fixed to ground-truth values (or freeze all three streams to get ground-truth correlation values); + * The parameters `T-f0`, `T-duration`, and `T-token` specify per-stream temperatures and, in the case of continuous-valued prosody, scaling parameter of the corresponding Laplace distribution (setting a temperature to 0 will enforce greedy sampling); + * `min-length` filters out sequences that are shorter then 300 duration units (i.e. 6s in the case of Hubert units); + * `prefix-length` specifies that we want to use first 150 duration units are prompt (i.e. 3s in the case of Hubert units) + + +**Correctness (aka Continuation) and Expressiveness (aka Std) metrics** + +By running the following command, we can get minMAE and Std for the log-F0 stream for the model with quantized prosody. +```bash +DATA=data_config.json +EXPLOSION=20 +SET=test +CHECKPOINT_PATH=discrete_prosody_shift_1_1.pt +T_F0=0.7 + +python examples/textless_nlp/pgslm/eval/cont_metrics.py $DATA \ + --prefix-length=150 \ + --metric=continuation \ + --path=$CHECKPOINT_PATH \ + --batch-size=16 \ + --fp16 \ + --seed=111 \ + --batch-explosion-rate=$EXPLOSION \ + --teacher-force-tokens \ + --teacher-force-duration \ + --T-f0=$T_F0 \ + --eval-subset=$SET \ + --f0-discretization-bounds=mean_norm_log_f0_seg_bin.th --dequantize-prosody +``` +(Using this command, our provided `discrete_prosody_shift_1_1.pt` checkpoint should produce `{...'F0 MAE': 0.0772, 'F0 Std': 0.1489...}` on LibriSpeech test-clean). + +Again, by setting `--teacher-force-tokens, --teacher-force-duration, --teacher-force-f0` we can calculate Token BLEU for the token stream (when `--teacher-force-duration` & `--teacher-force-f0` are on) and per-stream min MAE for each prosody stream individually. + +Finally, `cont_metrics.py` allows to specify the number of workers (e.g., `n-workers=8`) which allows to speed up the computation by spreading multiple worker processes +over the available GPUs. + +**Cont Word BLEU** + +We used the code and the evaluation protocol of [(Lakhotia et al., 2021)](https://arxiv.org/abs/2102.01192). + +## Sampling from a trained model + +To get (prompted or not) samples from a trained model it is enough to run `sample.py`: +```bash +CHECKPOINT_PATH=checkpoints/checkpoint_best.pt +DATASET=examples/textless_nlp/pgslm/repro/dataset/data_config.json +python examples/textless_nlp/pgslm/sample/sample.py $DATASET \ + --output=$SAMPLES \ + --path=$CHECKPOINT_PATH \ + --sampling \ + --T-token=0.7 \ + --T-duration=0.25 \ + --T-f0=0.7 \ + --max-length=500 \ + --prefix-length=150 \ + --subset=valid \ + --seed=1 \ + --match-duration \ + --code-type=hubert \ + --batch-explosion-rate=2 +``` + +Some useful parameters: + * `T-token`, `T-duration`, `T-f0` specify sampling temperature for the three streams. Setting a temperature to `0` switches sample to the greedy (argmax) one; + * `prefix-length`: length of the prompt, measured in timesteps (e.g. for Hubert (CPC) each timestep is 20 (10) ms); + * `subset`: which subset of the dataset to use as prompts (can be `train`, `valid`, `test`); + * `teacher-force-tokens`, `teacher-force-duration`, `teacher-force-f0`: if set, at each autoregressive step, ground-truth values replace the produced one; + * `short-curcuit`: replace sampling by ground-truth inputs; + * `match-duration`: forces the produced sample to have the same duration (in time), as the entire sequence (beyond the prompt if there is any); + * `batch-explosion-rate`: number of samples per prompt; + * `f0-discretization-bounds`: path to a file with quantization boundaries. If it is set, F0 values are de-quantized back to the continuous domain + (the model must be a quanized one); + * `max-length` sets the maximal number of segment steps to be produced. + +Note that `sample.py` automatically uses all available GPUs, to avoid that please use environment variable `CUDA_VISIBLE_DEVICES`. + +## Vocoding samples +To generate audios for output from `sample.py` (`$IN_FILE`): +```bash +python examples/textless_nlp/pgslm/generate_waveform.py \ + --in-file=$IN_FILE \ + --vocoder=$VODOER \ + --vocoder-cfg=$VOCODER_CFG \ + --results-path=$RESULTS_PATH +``` +See "Pre-trained model" for `$VOCODER` and `VOCODER_CFG`. diff --git a/data/fairseq/examples/textless_nlp/pgslm/data_utils.py b/data/fairseq/examples/textless_nlp/pgslm/data_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..2033697b372ce615b184589a8b3c4212798f5073 --- /dev/null +++ b/data/fairseq/examples/textless_nlp/pgslm/data_utils.py @@ -0,0 +1,107 @@ +# Copyright (c) Facebook, Inc. and its affiliates. + +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import os +import torch + +from tqdm import tqdm + + +class Stat: + def __init__(self, keep_raw=False): + self.x = 0.0 + self.x2 = 0.0 + self.z = 0.0 # z = logx + self.z2 = 0.0 + self.n = 0.0 + self.u = 0.0 + self.keep_raw = keep_raw + self.raw = [] + + def update(self, new_x): + new_z = new_x.log() + + self.x += new_x.sum() + self.x2 += (new_x**2).sum() + self.z += new_z.sum() + self.z2 += (new_z**2).sum() + self.n += len(new_x) + self.u += 1 + + if self.keep_raw: + self.raw.append(new_x) + + @property + def mean(self): + return self.x / self.n + + @property + def std(self): + return (self.x2 / self.n - self.mean**2) ** 0.5 + + @property + def mean_log(self): + return self.z / self.n + + @property + def std_log(self): + return (self.z2 / self.n - self.mean_log**2) ** 0.5 + + @property + def n_frms(self): + return self.n + + @property + def n_utts(self): + return self.u + + @property + def raw_data(self): + assert self.keep_raw, "does not support storing raw data!" + return torch.cat(self.raw) + + +class F0Stat(Stat): + def update(self, new_x): + # assume unvoiced frames are 0 and consider only voiced frames + if new_x is not None: + super().update(new_x[new_x != 0]) + + +def dump_speaker_f0_stat(speaker_to_f0_stat, out_prefix): + path = f"{out_prefix}.f0_stat.pt" + assert not os.path.exists(path) + + d = { + speaker: { + "f0_mean": speaker_to_f0_stat[speaker].mean, + "f0_std": speaker_to_f0_stat[speaker].std, + "logf0_mean": speaker_to_f0_stat[speaker].mean_log, + "logf0_std": speaker_to_f0_stat[speaker].std_log, + } + for speaker in speaker_to_f0_stat + } + torch.save(d, path) + + return d + + +def load_audio_path(path): + audio_paths = [] + with open(path) as f: + for line in f.readlines(): + sample = eval(line.strip()) + audio_paths.append(sample["audio"]) + + return audio_paths + + +def load_f0(f0_dir, nshards): + path_to_f0 = {} + for rank in tqdm(range(1, nshards + 1), desc=f"load f0"): + f0_shard_path = f"{f0_dir}/f0_{rank}_{nshards}.pt" + shard_path_to_f0 = torch.load(f0_shard_path) + path_to_f0.update(shard_path_to_f0) + return path_to_f0 diff --git a/data/fairseq/examples/textless_nlp/pgslm/eval/__init__.py b/data/fairseq/examples/textless_nlp/pgslm/eval/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..0e028c26b9abf533d16fa07d00d942ca07bbba61 --- /dev/null +++ b/data/fairseq/examples/textless_nlp/pgslm/eval/__init__.py @@ -0,0 +1,4 @@ +# Copyright (c) Facebook, Inc. and its affiliates. + +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. diff --git a/data/fairseq/examples/textless_nlp/pgslm/eval/cont_metrics.py b/data/fairseq/examples/textless_nlp/pgslm/eval/cont_metrics.py new file mode 100644 index 0000000000000000000000000000000000000000..e98abadde3f4c22adfe6ef3a5d9bd68c4e62d408 --- /dev/null +++ b/data/fairseq/examples/textless_nlp/pgslm/eval/cont_metrics.py @@ -0,0 +1,730 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import os +import numpy as np +import scipy + +import torch +import torch.multiprocessing as mp +from fairseq import checkpoint_utils, options +from fairseq.data.codedataset import CodeDataset, ExpressiveCodeDataConfig +from fairseq.dataclass.utils import convert_namespace_to_omegaconf +from torch.utils.data import DataLoader, DistributedSampler +from fairseq.utils import move_to_cuda +from fairseq import utils +from fairseq.criterions.speech_ulm_criterion import nll_loss, mae_loss + +import time +from types import SimpleNamespace + +import sys, pathlib + +sys.path.append(str(pathlib.Path(__file__).parent.parent.resolve())) + +from naive_decoder import Naive_F0_Decoder +from inference_dataset import InferenceDataset, explode_batch +from sample.sample import do_sampling, TemperatureDecoder, FilterNamesDataset + +try: + from nltk.translate.bleu_score import sentence_bleu +except ImportError: + print("Please install nltk: `pip install --user -U nltk`") + raise + + +@torch.no_grad() +def teacher_force_everything( + args, dataset, model, criterion, tgt_dict, rank, world_size +): + prefix = args.prefix_length + + f0_decoder = None + if args.dequantize_prosody: + assert dataset.discrete_f0 + print("Reporting MAE for a discrete model") + f0_decoder = Naive_F0_Decoder( + args.f0_discretization_bounds, dataset.config.f0_vq_n_units + ).cuda() + + dataset = InferenceDataset( + dataset, + prefix=args.prefix_length, + only_prefix=False, + filter_short=True, + presort_by_length=True, + ) + sampler = ( + None + if world_size == 1 + else DistributedSampler( + dataset, num_replicas=world_size, rank=rank, shuffle=False + ) + ) + dataloader = DataLoader( + dataset, + args.batch_size, + shuffle=False, + collate_fn=dataset.collater, + sampler=sampler, + ) + + total_token_loss, total_duration_loss, total_f0_loss, total_tokens = ( + 0.0, + 0.0, + 0.0, + 0.0, + ) + + i = 0 + for batch in dataloader: + i += 1 + batch = move_to_cuda(batch) + output = model(**batch["net_input"]) + + tokens, durations, f0 = output["token"], output["duration"], output["f0"] + durations, f0 = durations.squeeze(), f0.squeeze() + + token_loss = nll_loss( + tokens[:, prefix - 1 :], + batch["target"][:, prefix - 1 :].contiguous(), + batch["mask"][:, prefix - 1 :].contiguous(), + reduce=True, + ) + + if args.dequantize_prosody: + durations = durations.argmax(dim=-1) + duration_loss = mae_loss( + durations[:, prefix - 1 :].contiguous().float(), + batch["dur_target"][:, prefix - 1 :].contiguous().float(), + batch["dur_mask"][:, prefix - 1 :].contiguous(), + reduce=True, + ) + else: + duration_loss = criterion.dur_loss_fn( + durations[:, prefix - 1 :].contiguous(), + batch["dur_target"][:, prefix - 1 :].contiguous(), + batch["dur_mask"][:, prefix - 1 :].contiguous(), + reduce=True, + ) + + if f0_decoder: + f0 = f0.argmax(dim=-1) + f0 = f0_decoder(f0).squeeze(-1) + + f0_target = batch["raw_f0"] + f0_loss = mae_loss( + f0[:, prefix - 1 :].contiguous(), + f0_target[:, prefix - 1 :].contiguous(), + batch["f0_mask"][:, prefix - 1 :].contiguous(), + reduce=True, + ) + else: + f0_loss = criterion.f0_loss_fn( + f0[:, prefix - 1 :].contiguous(), + batch["f0_target"][:, prefix - 1 :].contiguous(), + batch["f0_mask"][:, prefix - 1 :].contiguous(), + reduce=True, + ) + + n_tokens = (~batch["dur_mask"])[:, prefix - 1 :].sum() + + total_token_loss += token_loss.item() + total_duration_loss += duration_loss.item() + total_f0_loss += f0_loss.item() + + total_tokens += n_tokens.item() + if args.debug and i > 5: + break + + values = torch.tensor([total_token_loss, total_duration_loss, total_f0_loss]) + normalizers = torch.tensor([total_tokens for _ in range(3)]) + + return values, normalizers + + +def get_bleu(produced_tokens, target_tokens, tgt_dict): + assert target_tokens.ndim == 1 + assert produced_tokens.size(1) == target_tokens.size(0) + + # we can have padding due to shifted channels + shift = 0 + for token in reversed(target_tokens.cpu().tolist()): + if token in [tgt_dict.pad(), tgt_dict.eos()]: + shift += 1 + else: + break + target_tokens = target_tokens[:-shift] + produced_tokens = produced_tokens[:, :-shift] + + string_target = tgt_dict.string(target_tokens).split() + string_candidates = [ + tgt_dict.string(produced_tokens[i, :]).split() + for i in range(produced_tokens.size(0)) + ] + + bleu3 = sentence_bleu( + references=string_candidates, + hypothesis=string_target, + weights=(1.0 / 3, 1.0 / 3, 1.0 / 3), + ) + return bleu3 + + +@torch.no_grad() +def continuation(args, dataset, model, criterion, tgt_dict, rank, world_size): + is_discrete_duration = dataset.discrete_dur + is_discrete_f0 = dataset.discrete_f0 + + f0_decoder = None + if args.dequantize_prosody: + assert dataset.discrete_f0 + print("Reporting MAE F0 for a discrete model") + f0_decoder = Naive_F0_Decoder( + args.f0_discretization_bounds, dataset.config.f0_vq_n_units + ).cuda() + + dataset = InferenceDataset( + dataset, args.prefix_length, filter_short=True, presort_by_length=True + ) + sampler = ( + None + if world_size == 1 + else DistributedSampler( + dataset, num_replicas=world_size, rank=rank, shuffle=False + ) + ) + dataloader = DataLoader( + dataset, + batch_size=1, + shuffle=False, + collate_fn=dataset.collater, + sampler=sampler, + ) + + Ts = args.T_token, args.T_duration, args.T_f0 + decoder = TemperatureDecoder( + Ts, discrete_dur=is_discrete_duration, discrete_f0=is_discrete_f0 + ) + + running_stats = SimpleNamespace( + token_bleu=0.0, + duration_nll=0.0, + duration_mae=0.0, + f0_nll=0.0, + f0_mae=0.0, + n_tokens=0.0, + n_sentences=0.0, + f0_sum=0.0, + f0_sum_sq=0.0, + dur_sum=0.0, + dur_sum_sq=0.0, + ) + + for i, batch in enumerate(dataloader): + batch = explode_batch(batch, args.batch_explosion_rate) + bsz = batch["target"].size(0) + + batch = move_to_cuda(batch) + prefix = batch["prefix"][0] + + max_length_to_unroll = batch["target"].size(1) + prefix_length = batch["net_input"]["src_tokens"].size(1) + steps = max_length_to_unroll - prefix_length + 1 + + assert steps > 0 + produced_tokens, produced_durations, produced_f0, outputs = do_sampling( + model, + batch, + tgt_dict.eos(), + decoder, + autoregressive_steps=steps, + teacher_force_tokens=args.teacher_force_tokens, + teacher_force_duration=args.teacher_force_duration, + teacher_force_f0=args.teacher_force_f0, + ) + + if args.teacher_force_tokens: + assert (produced_tokens[:, 1:] == batch["target"]).all() + if args.teacher_force_duration: + assert (produced_durations[:, 1:] == batch["dur_target"]).all() + if args.teacher_force_f0: + assert (produced_f0[:, 1:] == batch["f0_target"]).all() + + dur_target = batch["dur_target"][:, prefix - 1 :].contiguous() + f0_target = batch["f0_target"][:, prefix - 1 :].contiguous() + + f0_mask = batch["f0_mask"][:, prefix - 1 :].contiguous() + dur_mask = batch["dur_mask"][:, prefix - 1 :].contiguous() + + duration_mae = mae_loss( + produced_durations[:, prefix:].float(), + dur_target.float(), + dur_mask, + reduce=False, + ) + min_duration_mae = duration_mae.view(bsz, -1).sum(dim=-1).min(dim=0)[0] + running_stats.duration_mae += min_duration_mae + + running_stats.dur_sum += ( + produced_durations[:, prefix:].float() * (~dur_mask) + ).sum() / args.batch_explosion_rate + running_stats.dur_sum_sq += ( + produced_durations[:, prefix:].float() * (~dur_mask) + ).pow(2.0).sum() / args.batch_explosion_rate + + if is_discrete_duration: + duration_loss = criterion.dur_loss_fn( + torch.stack([x[1] for x in outputs], dim=1), + dur_target, + dur_mask, + reduce=False, + ) + min_duration_loss = duration_loss.view(bsz, -1).sum(dim=-1).min(dim=0)[0] + running_stats.duration_nll += min_duration_loss + + if f0_decoder: # can only exist for discrete F0 models + decoded_produced_f0 = f0_decoder(produced_f0[:, prefix:]) + decoded_f0_target = batch["raw_f0"][:, prefix - 1 :].contiguous() + + if produced_f0.ndim == 3: + decoded_produced_f0 = decoded_produced_f0.squeeze(2) + decoded_f0_target = decoded_f0_target.squeeze(2) + + f0_mae = mae_loss( + decoded_produced_f0, decoded_f0_target, f0_mask, reduce=False + ) + f0_mae = f0_mae.view(bsz, -1).sum(dim=-1).min(dim=0)[0] + running_stats.f0_mae += f0_mae + + f0_loss = criterion.f0_loss_fn( + torch.stack([x[2] for x in outputs], dim=1), + f0_target.long(), + f0_mask, + reduce=False, + ) + f0_loss = f0_loss.view(bsz, -1).sum(dim=-1).min(dim=0)[0] + running_stats.f0_nll += f0_loss + + running_stats.f0_sum += ( + decoded_produced_f0 * (~f0_mask) + ).sum() / args.batch_explosion_rate + running_stats.f0_sum_sq += (decoded_produced_f0 * (~f0_mask)).pow( + 2.0 + ).sum() / args.batch_explosion_rate + + else: + assert not is_discrete_duration + + f0_loss = mae_loss( + produced_f0[:, prefix:], f0_target, f0_mask, reduce=False + ) + f0_loss = f0_loss.view(bsz, -1).sum(dim=-1).min(dim=0)[0] + running_stats.f0_mae += f0_loss + + running_stats.f0_sum += ( + produced_f0[:, prefix:].sum() / args.batch_explosion_rate + ) + running_stats.f0_sum_sq += ( + produced_f0[:, prefix:].pow(2.0).sum() / args.batch_explosion_rate + ) + + running_stats.n_tokens += (~dur_mask)[0, ...].sum() + + token_loss = get_bleu( + produced_tokens[:, prefix:], batch["target"][0, prefix - 1 :], tgt_dict + ) + running_stats.token_bleu += token_loss + running_stats.n_sentences += 1 + + if args.debug: + break + + values = torch.tensor( + [ + running_stats.token_bleu, + running_stats.duration_nll, + running_stats.duration_mae, + running_stats.f0_nll, + running_stats.f0_mae, + running_stats.f0_sum, + running_stats.f0_sum_sq, + running_stats.dur_sum, + running_stats.dur_sum_sq, + ] + ) + normalizers = torch.tensor( + [running_stats.n_sentences] + [running_stats.n_tokens] * 8 + ) + + return values, normalizers + + +@torch.no_grad() +def correlation(args, dataset, model, criterion, tgt_dict, rank, world_size): + is_discrete_duration = dataset.discrete_dur + is_discrete_f0 = dataset.discrete_f0 + + f0_decoder = None + if is_discrete_f0: + assert dataset.discrete_f0 + f0_decoder = Naive_F0_Decoder( + args.f0_discretization_bounds, dataset.config.f0_vq_n_units + ).cuda() + + if is_discrete_f0: + assert f0_decoder # correlation on tokens is meaningless + + dataset = InferenceDataset( + dataset, + args.prefix_length, + filter_short=True, + presort_by_length=True, + min_length=args.min_length, + ) + sampler = ( + None + if world_size == 1 + else DistributedSampler( + dataset, num_replicas=world_size, rank=rank, shuffle=False + ) + ) + dataloader = DataLoader( + dataset, + batch_size=1, + shuffle=False, + collate_fn=dataset.collater, + sampler=sampler, + ) + + Ts = args.T_token, args.T_duration, args.T_f0 + decoder = TemperatureDecoder( + Ts, discrete_dur=is_discrete_duration, discrete_f0=is_discrete_f0 + ) + + mean_dur_prefix, mean_dur_cont = [], [] + mean_f0_prefix, mean_f0_cont = [], [] + + for batch in dataloader: + batch = explode_batch(batch, args.batch_explosion_rate) + batch = move_to_cuda(batch) + + assert len(batch["prefix"]) == 1 + + if args.teacher_force_tokens: + autoregressive_steps = batch["target"].size(1) - args.prefix_length - 1 + else: + autoregressive_steps = args.max_length - args.prefix_length # + max_shift? + + if args.copy_target: + produced_durations, produced_f0 = batch["dur_target"], batch["f0_target"] + else: + _, produced_durations, produced_f0, outputs = do_sampling( + model, + batch, + tgt_dict.eos(), + decoder, + autoregressive_steps=autoregressive_steps, + teacher_force_tokens=args.teacher_force_tokens, + teacher_force_duration=args.teacher_force_duration, + teacher_force_f0=args.teacher_force_f0, + ) + + # first tokens actually correspond to BOS + produced_durations = produced_durations[:, 1:] + produced_f0 = produced_f0[:, 1:] + + dur_target = batch["dur_target"] + if is_discrete_duration: + produced_durations = produced_durations.float() + dur_target = dur_target.float() + + if is_discrete_f0: + produced_f0 = f0_decoder(produced_f0).squeeze(-1) + f0_target = batch["raw_f0"] + else: + f0_target = batch["f0_target"] + + # prefix values + prefix = batch["prefix"][0] + dur_prefix_mean = dur_target[:, :prefix].sum(dim=-1) / ( + (~batch["dur_mask"][:, :prefix]).sum(dim=-1) + ) + + non_voiced = f0_target[:, :prefix] == 0.0 + f0_mask = batch["f0_mask"][:, :prefix].logical_or(non_voiced) + f0_prefix_mean = f0_target[:, :prefix].sum(dim=-1) / ((~f0_mask).sum(dim=-1)) + + # continuation values + dur_cont_mean = produced_durations[:, prefix:].sum(dim=-1) / ( + (~batch["dur_mask"][:, prefix:]).sum(dim=-1) + ) + + non_voiced = produced_f0[:, prefix:] == 0.0 + f0_mask = non_voiced + f0_cont_mean = produced_f0[:, prefix:].sum(dim=-1) / ((~f0_mask).sum(dim=-1)) + + assert not f0_cont_mean.isnan().any() + + mean_dur_prefix.append(dur_prefix_mean.cpu()) + mean_dur_cont.append(dur_cont_mean.cpu()) + + mean_f0_prefix.append(f0_prefix_mean.cpu()) + mean_f0_cont.append(f0_cont_mean.cpu()) + + if args.debug and len(mean_dur_prefix) > 10: + break + + mean_dur_prefix, mean_dur_cont = torch.cat(mean_dur_prefix), torch.cat( + mean_dur_cont + ) + mean_f0_prefix, mean_f0_cont = torch.cat(mean_f0_prefix), torch.cat(mean_f0_cont) + + return mean_dur_prefix, mean_dur_cont, mean_f0_prefix, mean_f0_cont + + +def main(rank, world_size, args): + start = time.time() + + if world_size > 1: + torch.distributed.init_process_group( + backend="gloo", init_method="env://", world_size=world_size, rank=rank + ) + torch.cuda.set_device(rank % torch.cuda.device_count()) + + raw_args = args + + args = convert_namespace_to_omegaconf(args) + if args.common.seed is not None: + np.random.seed(args.common.seed) + utils.set_torch_seed(args.common.seed) + + models, model_args, task = checkpoint_utils.load_model_ensemble_and_task( + [raw_args.path], arg_overrides={"data": args.task.data} + ) + + tgt_dict = task.target_dictionary + + for model in models: + model.prepare_for_inference_(args) + model.cuda().eval() + if raw_args.fp16: + model = model.half() + model = models[0] + + config = ExpressiveCodeDataConfig(args.task.data) + + dataset = CodeDataset( + manifest=config.manifests[raw_args.eval_subset], + dictionary=task.source_dictionary, + dur_dictionary=task.source_duration_dictionary, + f0_dictionary=task.source_f0_dictionary, + config=config, + discrete_dur=task.cfg.discrete_duration, + discrete_f0=task.cfg.discrete_f0, + log_f0=task.cfg.log_f0, + normalize_f0_mean=task.cfg.normalize_f0_mean, + normalize_f0_std=task.cfg.normalize_f0_std, + interpolate_f0=task.cfg.interpolate_f0, + shifts=task.cfg.stream_shifts, + return_filename=True, + strip_filename=False, + return_continuous_f0=raw_args.dequantize_prosody, + ) + + if raw_args.filter_names: + dataset = FilterNamesDataset(dataset, raw_args.filter_names) + + criterion = task.build_criterion(model_args.criterion) + + name2metric = { + "continuation": continuation, + "teacher_force_everything": teacher_force_everything, + "correlation": correlation, + } + + name2keys = { + "continuation": ( + "Token BLEU3", + "Duration NLL", + "Duration MAE", + "F0 NLL", + "F0 MAE", + "F0 sum", + "F0 sum_sq", + "Dur sum", + "Dur sum_sq", + ), + "teacher_force_everything": ("token_loss", "duration_loss", "f0_loss"), + "correlation": ("Duration corr", "F0 corr"), + } + metric_name = raw_args.metric + + metric = name2metric[metric_name] + results = metric(raw_args, dataset, model, criterion, tgt_dict, rank, world_size) + + values = None + + if metric_name not in [ + "correlation", + ]: + values, normalizers = results + values = maybe_aggregate_normalize(values, normalizers, world_size) + elif metric_name == "correlation": + values = maybe_aggregate_correlations(results, world_size) + else: + assert False + + assert values is not None + summary = dict(zip(name2keys[raw_args.metric], values.tolist())) + if metric_name == "continuation": + summary["F0 Std"] = np.sqrt(-summary["F0 sum"] ** 2 + summary["F0 sum_sq"]) + summary["Dur Std"] = np.sqrt(-summary["Dur sum"] ** 2 + summary["Dur sum_sq"]) + del summary["F0 sum"] + del summary["F0 sum_sq"] + del summary["Dur sum"] + del summary["Dur sum_sq"] + + summary["metric"] = metric_name + + if rank == 0: + print(summary) + if raw_args.wandb: + wandb_results(summary, raw_args) + print("# finished in ", time.time() - start, "seconds") + + +def wandb_results(summary, raw_args): + import wandb + + run = wandb.init( + project=raw_args.wandb_project_name, tags=raw_args.wandb_tags.split(",") + ) + run.config.metric = raw_args.metric + run.config.model = raw_args.path + run.config.data = raw_args.data + + if raw_args.wandb_run_name: + run.name = raw_args.wandb_run_name + run.save() + + wandb.log(summary) + wandb.finish() + + +def maybe_aggregate_normalize(values, normalizers, world_size): + if world_size > 1: + torch.distributed.barrier() + + torch.distributed.all_reduce_multigpu([values]) + torch.distributed.all_reduce_multigpu([normalizers]) + + return values / normalizers + + +def maybe_aggregate_correlations(results, world_size): + if world_size > 1: + output = [None for _ in range(world_size)] + torch.distributed.all_gather_object(output, results) + mean_dur_prefix, mean_dur_cont, mean_f0_prefix, mean_f0_cont = [ + torch.cat([x[i] for x in output]) for i in range(4) + ] + else: + mean_dur_prefix, mean_dur_cont, mean_f0_prefix, mean_f0_cont = results + + corr_dur = scipy.stats.pearsonr(mean_dur_prefix.numpy(), mean_dur_cont.numpy())[0] + corr_f0 = scipy.stats.pearsonr(mean_f0_prefix.numpy(), mean_f0_cont.numpy())[0] + values = torch.tensor([corr_dur, corr_f0]) + + return values + + +def cli_main(): + parser = options.get_interactive_generation_parser() + parser.add_argument( + "--prefix-length", + type=int, + default=1, + help="Prompt prefix length (including )", + ) + parser.add_argument( + "--duration-scale", + type=float, + default=1, + help="Multiply durations by the given scaler", + ) + parser.add_argument( + "--debug", action="store_true", help="Process only the first batch" + ) + parser.add_argument("--n_hypotheses", type=int, default=1) + parser.add_argument("--filter-names", type=str, default=None) + parser.add_argument( + "--max-length", type=int, default=200, help="Maximal produced length" + ) + + parser.add_argument("--teacher-force-tokens", action="store_true", default=False) + parser.add_argument("--teacher-force-duration", action="store_true", default=False) + parser.add_argument("--teacher-force-f0", action="store_true", default=False) + + parser.add_argument("--copy-target", action="store_true", default=False) + parser.add_argument("--min-length", type=int, default=None) + parser.add_argument("--f0-discretization-bounds", type=str, default=None) + parser.add_argument("--dequantize-prosody", action="store_true") + parser.add_argument("--batch-explosion-rate", type=int, default=1) + + parser.add_argument( + "--metric", + choices=["continuation", "teacher_force_everything", "correlation"], + required=True, + ) + + parser.add_argument("--wandb", action="store_true") + parser.add_argument("--wandb-project-name", type=str, default="eslm") + parser.add_argument("--wandb-tags", type=str, default="") + parser.add_argument("--wandb-run-name", type=str, default="") + + parser.add_argument("--T-token", type=float, default=1.0) + parser.add_argument("--T-duration", type=float, default=1.0) + parser.add_argument("--T-f0", type=float, default=1.0) + + parser.add_argument("--n-workers", type=int, default=1) + + parser.add_argument( + "--eval-subset", type=str, default="valid", choices=["valid", "test"] + ) + + args = options.parse_args_and_arch(parser) + + assert ( + args.prefix_length >= 1 + ), "Prefix length includes bos token , hence the minimum is 1." + assert args.temperature >= 0.0, "T must be non-negative!" + + if args.dequantize_prosody: + assert args.f0_discretization_bounds + + world_size = args.n_workers or torch.cuda.device_count() + if world_size > 1: + import random + + mp.set_start_method("spawn", force=True) + os.environ["MASTER_ADDR"] = "localhost" + os.environ["MASTER_PORT"] = str(random.randint(10_000, 50_000)) + + mp.spawn( + main, + nprocs=world_size, + args=( + world_size, + args, + ), + join=True, + ) + else: + main(rank=0, world_size=world_size, args=args) + + +if __name__ == "__main__": + cli_main() diff --git a/data/fairseq/examples/textless_nlp/pgslm/generate_waveform.py b/data/fairseq/examples/textless_nlp/pgslm/generate_waveform.py new file mode 100644 index 0000000000000000000000000000000000000000..a6f348bb9b3e3ee998a5f8e2dcafe01a45476c98 --- /dev/null +++ b/data/fairseq/examples/textless_nlp/pgslm/generate_waveform.py @@ -0,0 +1,120 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import ast +import argparse +import json +import logging +from pathlib import Path +import soundfile as sf +import torch + +from tqdm import tqdm + +from fairseq import utils +from fairseq.models.text_to_speech.vocoder import CodeHiFiGANVocoder + + +logging.basicConfig() +logging.root.setLevel(logging.INFO) +logging.basicConfig(level=logging.INFO) +logger = logging.getLogger(__name__) + + +def dump_result(args, data, sample_id, pred_wav): + assert "audio" in data or args.results_path is not None + if args.results_path: + fname = Path(data["audio"]).name if "audio" in data else f"{sample_id}_pred.wav" + out_file = Path(args.results_path) / fname + + sf.write( + out_file.as_posix(), + pred_wav.detach().cpu().numpy(), + args.sample_rate, + ) + + +def load_data(in_file): + with open(in_file) as f: + data = [ast.literal_eval(line.strip()) for line in f] + + return data + + +def get_f0_upsample_ratio(code_hop_size, f_hop_size): + ratio = (code_hop_size // 160) // (f_hop_size // 256) * 2 + return ratio + + +def main(args): + logger.info(args) + + use_cuda = torch.cuda.is_available() and not args.cpu + + with open(args.vocoder_cfg) as f: + vocoder_cfg = json.load(f) + vocoder = CodeHiFiGANVocoder(args.vocoder, vocoder_cfg) + if use_cuda: + vocoder = vocoder.cuda() + + data = load_data(args.in_file) + + if args.results_path: + Path(args.results_path).mkdir(exist_ok=True, parents=True) + + for i, d in tqdm(enumerate(data), total=len(data)): + code_key = "cpc_km100" if "cpc_km100" in d else "hubert" + code = list(map(int, d[code_key].split())) + + x = { + "code": torch.LongTensor(code).view(1, -1), + "f0": torch.Tensor(d["f0"]).view(1, -1), + } + + f0_up_ratio = get_f0_upsample_ratio( + vocoder_cfg["code_hop_size"], vocoder_cfg["hop_size"] + ) + if f0_up_ratio > 1: + bsz, cond_length = x["f0"].size() + x["f0"] = x["f0"].unsqueeze(2).repeat(1, 1, f0_up_ratio).view(bsz, -1) + + x = utils.move_to_cuda(x) if use_cuda else x + wav = vocoder(x) + dump_result(args, d, i, wav) + + +def cli_main(): + parser = argparse.ArgumentParser() + parser.add_argument( + "--in-file", + type=str, + required=True, + help="Input file following the same format of the output from sample.py ('f0' and 'cpc_km100/hubert' are required fields)", + ) + parser.add_argument( + "--vocoder", type=str, required=True, help="path to the vocoder" + ) + parser.add_argument( + "--vocoder-cfg", + type=str, + required=True, + help="path to the vocoder config", + ) + parser.add_argument("--sample-rate", type=int, default=16_000) + parser.add_argument( + "--results-path", + type=str, + default=None, + help="Output directory. If not set, the audios will be stored following the 'audio' field specified in the input file.", + ) + parser.add_argument("--cpu", action="store_true", help="run on CPU") + + args = parser.parse_args() + + main(args) + + +if __name__ == "__main__": + cli_main() diff --git a/data/fairseq/examples/textless_nlp/pgslm/inference_dataset.py b/data/fairseq/examples/textless_nlp/pgslm/inference_dataset.py new file mode 100644 index 0000000000000000000000000000000000000000..9f7cfa5f5437f17a665070f0c539c11a92e30855 --- /dev/null +++ b/data/fairseq/examples/textless_nlp/pgslm/inference_dataset.py @@ -0,0 +1,103 @@ +# Copyright (c) Facebook, Inc. and its affiliates. + +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + + +import torch + + +class InferenceDataset: + def __init__( + self, + dataset, + prefix, + only_prefix=True, + presort_by_length=True, + filter_short=False, + min_length=None, + ): + self.dataset = dataset + self.collater = self.dataset.collater + self.prefix = prefix + self.only_prefix = only_prefix + self.filter_short = filter_short + + self.remapping = list(range(len(self.dataset))) + if min_length: + assert min_length >= prefix + 1 + + length_thr = prefix + 1 if not min_length else min_length + + if filter_short: + self.remapping = list( + filter( + lambda i: self.dataset[i]["dur_source"].sum() > length_thr, + self.remapping, + ) + ) + print( + f"# the initial dataset of {len(self.dataset)} examples became {len(self.remapping)} after filtering" + f" examples shorter than {length_thr} (in duration units)" + ) + + if presort_by_length: + lengths = {index: dataset.size(index) for index in self.remapping} + self.remapping.sort(key=lambda i: lengths[i]) + + @property + def pads(self): + return self.dataset.pads + + def __len__(self): + return len(self.remapping) + + def original_size(self, k): + k = self.remapping[k] + return self.dataset.size(k) + + def __getitem__(self, k): + k = self.remapping[k] + channels = self.dataset[k] + + if self.prefix and self.only_prefix: + dur_channel = channels["dur_source"] + assert dur_channel.sum() >= self.prefix + + token_times = dur_channel.cumsum(dim=-1) + cut_after = torch.searchsorted(token_times, torch.tensor(self.prefix)) + + r = {} + for channel_name, value in channels.items(): + if isinstance(value, torch.Tensor) and "source" in channel_name: + # if self.filter_short: assert value.size(0) >= self.prefix + r[channel_name] = value[: cut_after + 1] + else: + r[channel_name] = value + + r["prefix"] = cut_after + 1 + else: + r = channels + + return r + + +def explode_batch(batch, times): + if times == 1: + return batch + + new_batch = {} + + for key, value in batch.items(): + if isinstance(value, torch.Tensor): + assert value.size(0) == 1 + new_batch[key] = torch.cat([value] * times) + elif key in ["ntokens", "nsentences"]: + new_batch[key] = value * times + elif key in ["prefix", "filename"]: + new_batch[key] = value + elif key == "net_input": + new_batch[key] = explode_batch(value, times) + else: + assert False, key + return new_batch diff --git a/data/fairseq/examples/textless_nlp/pgslm/naive_decoder.py b/data/fairseq/examples/textless_nlp/pgslm/naive_decoder.py new file mode 100644 index 0000000000000000000000000000000000000000..5132889792b7b2d073037a6b7f3003927d6a802a --- /dev/null +++ b/data/fairseq/examples/textless_nlp/pgslm/naive_decoder.py @@ -0,0 +1,40 @@ +# Copyright (c) Facebook, Inc. and its affiliates. + +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import torch +import warnings + + +class Naive_F0_Decoder(torch.nn.Module): + def __init__(self, bounds_path, n_units=32): + super().__init__() + + bounds = torch.load(bounds_path) + bounds = torch.from_numpy(bounds[n_units]) + assert bounds.ndim == 1 + + pad = torch.tensor([-5.0, -5.0]) # bos, eos, pad are in the dictionary + centers = torch.cat( + [bounds[0:1], 0.5 * (bounds[1:] + bounds[:-1]), bounds[-1:], pad[:]] + ) + + self.embedding = torch.nn.Embedding.from_pretrained( + centers.unsqueeze(-1), freeze=True + ) + self.max_n = self.embedding.weight.numel() + + def forward(self, discrete_f0: torch.Tensor): + in_bounds = (0 <= discrete_f0).all() and (discrete_f0 < self.max_n).all() + if not in_bounds: + warnings.warn( + f"F0 contains some weird outputs: discrete_f0.max().item()={discrete_f0.max().item()} discrete_f0.min().item()={discrete_f0.min().item()}; " + f"while we have embeddings for {self.max_n} values. " + "Assuming this is a no-prosody model -- but be careful!" + ) + + mask = discrete_f0 >= self.max_n + discrete_f0 = discrete_f0.masked_fill(mask, self.max_n - 1) + + return self.embedding(discrete_f0).squeeze(-1) diff --git a/data/fairseq/examples/textless_nlp/pgslm/prepare_dataset.py b/data/fairseq/examples/textless_nlp/pgslm/prepare_dataset.py new file mode 100644 index 0000000000000000000000000000000000000000..3d5edaa58fc701f614ee8494ad1f3cb2ad56bda6 --- /dev/null +++ b/data/fairseq/examples/textless_nlp/pgslm/prepare_dataset.py @@ -0,0 +1,143 @@ +# Copyright (c) Facebook, Inc. and its affiliates. + +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +from multiprocessing import Pool + +import os +from collections import defaultdict +from itertools import starmap + +import torch +from npy_append_array import NpyAppendArray +from tqdm import tqdm + +from data_utils import dump_speaker_f0_stat, F0Stat, load_f0 +from fairseq.data.codedataset import ( + ExpressiveCodeDataConfig, + parse_manifest, + F0_FRAME_SPACE, + align_f0_to_durations, +) +from fairseq.tasks.speech_ulm_task import UnitDictionary + + +def load_meta(meta_path, split): + config = ExpressiveCodeDataConfig(meta_path) + manifest_path = config.manifests[split] + dictionary = UnitDictionary(n_units=config.n_units) + audio_paths, codes, durs, speakers = parse_manifest(manifest_path, dictionary) + return config, audio_paths, codes, durs, speakers + + +def _align_f0(f0, dur, ratio, frm_tol=5): + if f0 is None: + seg_f0 = torch.zeros_like(dur, dtype=torch.float) + else: + seg_f0 = align_f0_to_durations(f0, dur, ratio, tol=frm_tol * ratio) + return seg_f0.numpy() # try a hacky stuff + + +def align_f0(path_to_f0, audio_paths, durs, ratio, mp=False): + chunk_size = 2000 + num_procs = 40 + iterable = ((path_to_f0[p], d, ratio) for p, d in zip(audio_paths, durs)) + + seg_f0s = [] + if mp: + with Pool(num_procs) as pool: + iterator = tqdm( + pool.istarmap(_align_f0, iterable, chunk_size), + desc="align f0", + total=len(durs), + ) + for seg_f0 in iterator: + seg_f0s.append(torch.from_numpy(seg_f0).float()) + else: + iterator = tqdm(starmap(_align_f0, iterable), desc="align f0", total=len(durs)) + for seg_f0 in iterator: + seg_f0s.append(torch.from_numpy(seg_f0).float()) + + return seg_f0s + + +def prepare_seg_data(config, audio_paths, codes, durs, speakers, path_to_f0): + ratio = config.code_hop_size / (config.sampling_rate * F0_FRAME_SPACE) + seg_f0s = align_f0(path_to_f0, audio_paths, durs, ratio) + data = { + "codes": codes, + "duration": durs, + "f0": seg_f0s, + "speaker": speakers, + "path": audio_paths, + } + return data + + +def dump_seg_data(data, out_prefix): + key_targs = { + "codes": f"{out_prefix}.code.npy", + "duration": f"{out_prefix}.dur.npy", + "f0": f"{out_prefix}.f0.npy", + } + for key, targ in key_targs.items(): + assert not os.path.exists(targ) + npaa = NpyAppendArray(targ) + for utt_data in tqdm(data[key], desc=f"dumping {key}"): + npaa.append(utt_data.numpy()) + + assert not os.path.exists(f"{out_prefix}.path.txt") + with open(f"{out_prefix}.path.txt", "w") as f: + for x in data["path"]: + f.write(f"{str(x)}\n") + + assert not os.path.exists(f"{out_prefix}.leng.txt") + with open(f"{out_prefix}.leng.txt", "w") as f: + for x in data["codes"]: + f.write(f"{len(x)}\n") + + assert not os.path.exists(f"{out_prefix}.speaker.txt") + with open(f"{out_prefix}.speaker.txt", "w") as f: + for x in data["speaker"]: + f.write(f"{str(x)}\n") + + print(f"wrote to files with prefix {out_prefix}") + + +def main(meta_path, f0_dir, splits, nshards_list): + speaker_to_stat = defaultdict(F0Stat) + if len(nshards_list) == 1: + nshards_list = nshards_list * len(splits) + else: + assert len(nshards_list) == len(splits) + + for split, nshards in zip(splits, nshards_list): + config, audio_paths, codes, durs, speakers = load_meta(meta_path, split) + path_to_f0 = load_f0(f"{f0_dir}/{split}", nshards) + + # segment-level data + data = prepare_seg_data(config, audio_paths, codes, durs, speakers, path_to_f0) + dump_seg_data(data, config.manifests[split]) + + # speaker f0 + for audio_path, speaker in tqdm(zip(audio_paths, speakers)): + f0 = path_to_f0[audio_path] + speaker_to_stat[speaker].update(f0) + dump_speaker_f0_stat(speaker_to_stat, config.manifests[split]) + + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser() + parser.add_argument("meta_path") + parser.add_argument("f0_dir", help="out_dir from preprocess_f0") + parser.add_argument("--splits", nargs="+", default=["train", "valid"]) + parser.add_argument( + "--nshards_list", type=int, nargs="+", default=[20], help="number of f0 shards" + ) + args = parser.parse_args() + print(args) + + main(**vars(args)) diff --git a/data/fairseq/examples/textless_nlp/pgslm/preprocess_f0.py b/data/fairseq/examples/textless_nlp/pgslm/preprocess_f0.py new file mode 100644 index 0000000000000000000000000000000000000000..afe899cb85f2fafe78e86cacba17f1d376c64394 --- /dev/null +++ b/data/fairseq/examples/textless_nlp/pgslm/preprocess_f0.py @@ -0,0 +1,65 @@ +# Copyright (c) Facebook, Inc. and its affiliates. + +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import os +import torch +from tqdm import tqdm +from data_utils import load_audio_path +from fairseq.data.codedataset import get_f0_by_filename + + +def process_one(path, sr): + """ + Args: + path: audio file path + sr: sampling rate + """ + try: + # YAAPT throws errors in some rare cases + f0 = get_f0_by_filename(path, sr) + except Exception as e: + print( + f"WARNING: error when processing {path}. set f0 to zero. original error message:\n{e}" + ) + f0 = None + return f0 + + +def main(file_path, out_dir, nshards, rank, sampling_rate): + # load data + audio_paths = load_audio_path(file_path) + + # shard + assert nshards <= len(audio_paths) and nshards > 0 + shard_size = len(audio_paths) / nshards + s = int(round((rank - 1) * shard_size)) + e = int(round(rank * shard_size)) + audio_paths = audio_paths[s:e] + + # process + path_to_f0 = {} + for i, audio_path in enumerate(tqdm(audio_paths)): + f0 = process_one(audio_path, sampling_rate) + path_to_f0[audio_path] = f0 + print(f"finished processing {len(path_to_f0)} utterances ({s}-{e})") + + f0_path = f"{out_dir}/f0_{rank}_{nshards}.pt" + os.makedirs(out_dir, exist_ok=True) + torch.save(path_to_f0, f0_path) + print(f"saved to {f0_path}") + + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser() + parser.add_argument("file_path") + parser.add_argument("out_dir") + parser.add_argument("--nshards", type=int, default=20) + parser.add_argument("--rank", type=int, default=1) + parser.add_argument("--sampling_rate", type=int, default=16000) + args = parser.parse_args() + + main(**vars(args)) diff --git a/data/fairseq/examples/textless_nlp/pgslm/quantize_f0.py b/data/fairseq/examples/textless_nlp/pgslm/quantize_f0.py new file mode 100644 index 0000000000000000000000000000000000000000..d9e3df2fe25f768a80f6db2d05d6b8eef436938d --- /dev/null +++ b/data/fairseq/examples/textless_nlp/pgslm/quantize_f0.py @@ -0,0 +1,94 @@ +# Copyright (c) Facebook, Inc. and its affiliates. + +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +from collections import defaultdict +from functools import partial + +import numpy as np +import torch +from tqdm import tqdm + +from data_utils import dump_speaker_f0_stat, F0Stat, load_audio_path, load_f0 + + +def load_speaker(path): + speakers = [] + with open(path) as f: + for line in f.readlines(): + sample = eval(line.strip()) + assert "speaker" in sample + speakers.append(sample["speaker"]) + return speakers + + +def quantize_f0(speaker_to_f0, f0_stats, nbins, normalize, log): + f0_all = [] + for speaker, f0 in speaker_to_f0.items(): + f0 = f0.raw_data + if log: + f0 = f0.log() + mean = f0_stats[speaker]["logf0_mean"] if log else f0_stats[speaker]["f0_mean"] + std = f0_stats[speaker]["logf0_std"] if log else f0_stats[speaker]["f0_std"] + if normalize == "mean": + f0 = f0 - mean + elif normalize == "meanstd": + f0 = (f0 - mean) / std + f0_all.extend(f0.tolist()) + + hist, bin_x = np.histogram(f0_all, 100000) + cum_hist = np.cumsum(hist) / len(f0_all) * 100 + + f0_bin = {} + for num_bin in nbins: + bin_offset = [] + bin_size = 100 / num_bin + threshold = bin_size + for i in range(num_bin - 1): + index = (np.abs(cum_hist - threshold)).argmin() + bin_offset.append(bin_x[index]) + threshold += bin_size + f0_bin[num_bin] = np.array(bin_offset) + + return f0_bin + + +def main(file_path, f0_dir, out_dir, out_prefix, nbins, nshards, normalize, log): + audio_paths = load_audio_path(file_path) + path_to_f0 = load_f0(f0_dir, nshards) + + speakers = load_speaker(file_path) + speaker_to_f0 = defaultdict(partial(F0Stat, True)) + + # speaker f0 stats + for audio_path, speaker in tqdm(zip(audio_paths, speakers)): + f0 = path_to_f0[audio_path] + speaker_to_f0[speaker].update(f0) + f0_stats = dump_speaker_f0_stat(speaker_to_f0, f"{out_dir}/{out_prefix}") + + # quantize + f0_bin = quantize_f0(speaker_to_f0, f0_stats, nbins, normalize, log) + log_suffix = "_log" if log else "" + f0_bin_out_file = f"{out_dir}/{out_prefix}_{normalize}_norm{log_suffix}_f0_bin.th" + torch.save(f0_bin, f0_bin_out_file) + + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser() + parser.add_argument("file_path") + parser.add_argument("f0_dir", help="out_dir from preprocess_f0") + parser.add_argument("out_dir") + parser.add_argument("out_prefix") + parser.add_argument("--nbins", nargs="+", type=int, default=[32]) + parser.add_argument("--nshards", type=int, default=20, help="number of f0 shards") + parser.add_argument( + "--normalize", type=str, choices=["meanstd", "mean", "none"], default="mean" + ) + parser.add_argument("--log", action="store_true") + args = parser.parse_args() + print(args) + + main(**vars(args)) diff --git a/data/fairseq/examples/textless_nlp/pgslm/sample/__init__.py b/data/fairseq/examples/textless_nlp/pgslm/sample/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..0e028c26b9abf533d16fa07d00d942ca07bbba61 --- /dev/null +++ b/data/fairseq/examples/textless_nlp/pgslm/sample/__init__.py @@ -0,0 +1,4 @@ +# Copyright (c) Facebook, Inc. and its affiliates. + +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. diff --git a/data/fairseq/examples/textless_nlp/pgslm/sample/sample.py b/data/fairseq/examples/textless_nlp/pgslm/sample/sample.py new file mode 100644 index 0000000000000000000000000000000000000000..55ec7a955e50073b858a263e4fe394d05895de71 --- /dev/null +++ b/data/fairseq/examples/textless_nlp/pgslm/sample/sample.py @@ -0,0 +1,612 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import os +import torch.multiprocessing as mp +import numpy as np +import json + +import torch +from torch.distributions.categorical import Categorical + +from fairseq import checkpoint_utils, options, utils +from fairseq.data.codedataset import CodeDataset, ExpressiveCodeDataConfig +from fairseq.dataclass.utils import convert_namespace_to_omegaconf +from torch.utils.data import DataLoader, DistributedSampler +from fairseq.utils import move_to_cuda + +import tqdm +import random +import pathlib + +import sys, pathlib + +sys.path.append(str(pathlib.Path(__file__).parent.parent)) +from inference_dataset import InferenceDataset, explode_batch +from naive_decoder import Naive_F0_Decoder +from truncated_laplace import truncated_laplace + +CODETYPE_TO_FRAMETIME = {"cpc_km100": 0.01, "hubert": 0.02} # 10ms # 20ms + + +class TemperatureDecoder: + def __init__(self, Ts, discrete_dur=False, discrete_f0=False): + self.T_token, self.T_dur, self.T_f0 = Ts + self.discrete_dur = discrete_dur + self.discrete_f0 = discrete_f0 + + def __call__(self, output): + def sample_multinomial(key, T): + logits = output[key][:, -1, :].float() + return Categorical(logits=logits / T).sample().unsqueeze(-1) + + def sample_laplace(key, T, truncate_at_zero): + mean = output[key][:, -1, :].float() + return truncated_laplace(mean=mean, T=T, truncate_by_zero=truncate_at_zero) + + if self.T_token > 0: + new_tokens = sample_multinomial("token", self.T_token) + else: + new_tokens = output["token"][:, -1, :].argmax(dim=-1, keepdim=True) + + if not self.discrete_dur and self.T_dur == 0: + new_durations = output["duration"][:, -1].round().int() + elif not self.discrete_dur and self.T_dur > 0: + new_durations = ( + sample_laplace("duration", self.T_dur, truncate_at_zero=True) + .round() + .int() + ) + elif self.discrete_dur and self.T_dur > 0: + new_durations = sample_multinomial("duration", self.T_dur) + elif self.discrete_dur and self.T_dur == 0: + new_durations = output["duration"][:, -1, :].argmax(dim=-1, keepdim=True) + else: + assert False + + if not self.discrete_f0 and self.T_f0 == 0: + new_f0 = output["f0"][:, -1] + elif not self.discrete_f0 and self.T_f0 > 0: + new_f0 = sample_laplace("f0", self.T_f0, truncate_at_zero=False) + elif self.discrete_f0 and self.T_f0 > 0: + new_f0 = sample_multinomial("f0", self.T_f0) + elif self.discrete_f0 and self.T_f0 == 0: + new_f0 = output["f0"][:, -1, :].argmax(dim=-1, keepdim=True) + else: + assert False + + return new_tokens, new_durations, new_f0 + + +class FilterNamesDataset: + def __init__(self, dataset, fnames_path): + self.dataset = dataset + + with open(fnames_path, "r") as fin: + fnames = set((eval(line)["audio"] for line in fin)) + print(f"# will retrict the dataset for {len(fnames)} files") + + self.indexes = [] + + for i, datapoint in enumerate(dataset): + if datapoint["filename"] in fnames: + self.indexes.append(i) + assert len(self.indexes) == len(fnames), f"{len(self.indexes)} {len(fnames)}" + + self.collater = self.dataset.collater + self.discrete_dur = self.dataset.discrete_dur + self.discrete_f0 = self.dataset.discrete_f0 + + def __len__(self): + return len(self.indexes) + + def __getitem__(self, k): + k = self.indexes[k] + return self.dataset[k] + + def size(self, k): + k = self.indexes[k] + return self.dataset.size(k) + + +@torch.no_grad() +def do_sampling( + model, + batch, + eos_token, + decoder, + autoregressive_steps=100, + teacher_force_tokens=False, + teacher_force_duration=False, + teacher_force_f0=False, + match_duration=False, +): + def autoregressive_step_(output, autoregressive_steps): + new_tokens, new_durations, new_f0 = decoder(output) + + n = output["token"].size(1) if output["token"].ndim == 3 else 1 + + if teacher_force_tokens: + new_tokens = batch["target"][:, n - 1].unsqueeze(-1) + if teacher_force_duration: + new_durations = batch["dur_target"][:, n - 1].unsqueeze(-1) + if teacher_force_f0: + new_f0 = batch["f0_target"][:, n - 1].unsqueeze(-1) + + batch["net_input"]["src_tokens"] = torch.cat( + [batch["net_input"]["src_tokens"], new_tokens], dim=1 + ) + batch["net_input"]["dur_src"] = torch.cat( + [batch["net_input"]["dur_src"], new_durations], dim=1 + ) + batch["net_input"]["f0_src"] = torch.cat( + [batch["net_input"]["f0_src"], new_f0], dim=1 + ) + + outputs = [] + + if teacher_force_tokens or teacher_force_duration or teacher_force_f0: + max_time = batch["target"].size(1) + prefix_time = batch["net_input"]["src_tokens"].size(1) + + autoregressive_steps = max_time - prefix_time + 1 # should be 0 + + for _ in range(autoregressive_steps): + output = model(**batch["net_input"]) + + last_steps = ( + output["token"][:, -1, ...], + output["duration"][:, -1, ...], + output["f0"][:, -1, ...], + ) + outputs.append(last_steps) + + autoregressive_step_(output, autoregressive_steps) + tokens, duration, f0 = ( + batch["net_input"]["src_tokens"], + batch["net_input"]["dur_src"], + batch["net_input"]["f0_src"], + ) + + if ( + match_duration + and (batch["dur_target"].sum(dim=-1) < duration.sum(dim=-1)).all() + ): + break + + return tokens, duration, f0, outputs + + +def unroll_duration(token_stream, duration_stream): + assert len(token_stream) == len( + duration_stream + ), f"{len(token_stream)} != {len(duration_stream)}" + non_positive_durations = sum(d <= 0 for d in duration_stream) + if non_positive_durations > 0: + print( + f"# {non_positive_durations} durations are non-positive, they will be capped to 1" + ) + + result = [] + + duration_stream_rounded_capped = [max(1, int(round(x))) for x in duration_stream] + for t, d in zip(token_stream, duration_stream_rounded_capped): + result.extend([t] * d) + + return result + + +def realign_shifted_streams(tokens, durations, F0s, shifts): + """ + Durations are shifted by 1, F0 by 2 + >>> tokens = ["", "t1", "t2", "t3", "", "x", "x"] + >>> durations = ["<0>", "<0>", "d1", "d2", "d3", "<0>", "x"] + >>> F0s = ["<0>", "<0>", "<0>", "f1", "f2", "f3", "<0>"] + >>> shifts = [1,2] + >>> realign_shifted_streams(tokens, durations, F0s, shifts) + (['', 't1', 't2', 't3', ''], ['<0>', 'd1', 'd2', 'd3', '<0>'], ['<0>', 'f1', 'f2', 'f3', '<0>']) + """ + max_shift = max(shifts) + if max_shift > 0: + shift_durations, shift_F0s = shifts + + tokens = tokens[:-max_shift] + durations = durations[shift_durations:] + if shift_durations < max_shift: + durations = durations[: -(max_shift - shift_durations)] + + if F0s is not None: + F0s = F0s[shift_F0s:] + if shift_F0s < max_shift: + F0s = F0s[: -(max_shift - shift_F0s)] + + assert len(tokens) == len(durations), f"{len(tokens)} =! {len(durations)}" + if F0s is not None: + assert len(tokens) == len(F0s), f"{len(tokens)} =! {len(F0s)}" + + return tokens, durations, F0s + + +def maybe_cut_eos(produced_tokens, produced_duration, produced_f0, eos_idx): + if eos_idx in produced_tokens: + eos_index = produced_tokens.index(eos_idx) + produced_tokens = produced_tokens[:eos_index] + produced_duration = produced_duration[:eos_index] + produced_f0 = produced_f0[:eos_index] + return produced_tokens, produced_duration, produced_f0 + + +def maybe_filter_pad(produced_tokens, produced_duration, produced_f0, pad_idx): + if pad_idx not in produced_tokens: + return produced_tokens, produced_duration, produced_f0 + + assert len(produced_tokens) == len(produced_duration) == len(produced_f0) + + print(" is detected in the output!") + filtered_tokens, filtered_duration, filtered_f0 = [], [], [] + + for t, d, f in zip(produced_tokens, produced_duration, produced_f0): + if t != pad_idx: + filtered_tokens.append(t) + filtered_duration.append(d) + filtered_f0.append(f) + return filtered_tokens, filtered_duration, filtered_f0 + + +def match_duration(produced_tokens, produced_duration, produced_f0, target_duration): + """ + >>> tokens = ['t'] * 4 + >>> F0s = ['f0'] * 4 + >>> produced_duration = [1, 10, 10, 10] + >>> match_duration(tokens, produced_duration, F0s, target_duration=100) + (['t', 't', 't', 't'], [1, 10, 10, 10], ['f0', 'f0', 'f0', 'f0']) + >>> match_duration(tokens, produced_duration, F0s, target_duration=5) + (['t', 't'], [1, 4], ['f0', 'f0']) + """ + if sum(produced_duration) <= target_duration: + return produced_tokens, produced_duration, produced_f0 + + running_duration = 0 + filtered_duration = [] + + for next_tok_duration in produced_duration: + if running_duration + next_tok_duration < target_duration: + filtered_duration.append(next_tok_duration) + running_duration += next_tok_duration + else: + to_add = target_duration - running_duration + assert to_add <= next_tok_duration + filtered_duration.append(to_add) + break + + produced_duration = filtered_duration + assert sum(produced_duration) == target_duration + + n_tok = len(filtered_duration) + + return produced_tokens[:n_tok], produced_duration, produced_f0[:n_tok] + + +def main(rank, world_size, args): + if world_size > 1: + torch.distributed.init_process_group( + backend="gloo", init_method="env://", world_size=world_size, rank=rank + ) + torch.cuda.set_device(rank) + + raw_args = args + args = convert_namespace_to_omegaconf(args) + if args.common.seed is not None: + random.seed(args.common.seed) + np.random.seed(args.common.seed) + utils.set_torch_seed(args.common.seed) + + models, model_args, task = checkpoint_utils.load_model_ensemble_and_task( + [raw_args.path], arg_overrides={"data": args.task.data} + ) + tgt_dict = task.target_dictionary + + for model in models: + model.prepare_for_inference_(args) + model.cuda().eval() + if raw_args.fp16: + model = model.half() + model = models[0] + + config = ExpressiveCodeDataConfig(args.task.data) + + dataset = CodeDataset( + manifest=config.manifests[raw_args.subset], + dictionary=task.source_dictionary, + dur_dictionary=task.source_duration_dictionary, + f0_dictionary=task.source_f0_dictionary, + config=config, + discrete_dur=task.cfg.discrete_duration, + discrete_f0=task.cfg.discrete_f0, + log_f0=task.cfg.log_f0, + normalize_f0_mean=task.cfg.normalize_f0_mean, + normalize_f0_std=task.cfg.normalize_f0_std, + interpolate_f0=task.cfg.interpolate_f0, + shifts=task.cfg.stream_shifts, + return_filename=True, + strip_filename=False, + ) + tgt_dict = task.target_dictionary + shifts = dataset.shifts.dur, dataset.shifts.f0 + max_shift = max(shifts) + + fname = raw_args.output + if world_size > 1: + fname += f"_{rank}" + output_file = open(fname, "w") + + if raw_args.filter_names: + dataset = FilterNamesDataset(dataset, raw_args.filter_names) + + dataset = InferenceDataset(dataset, raw_args.prefix_length, filter_short=True) + print(f"Dataset size {len(dataset)}") + sampler = ( + None + if world_size == 1 + else DistributedSampler( + dataset, num_replicas=world_size, rank=rank, shuffle=False + ) + ) + dataloader = DataLoader( + dataset, + batch_size=1, + shuffle=False, + collate_fn=dataset.collater, + sampler=sampler, + ) + + Ts = raw_args.T_token, raw_args.T_duration, raw_args.T_f0 + decoder = TemperatureDecoder( + Ts, discrete_dur=task.cfg.discrete_duration, discrete_f0=task.cfg.discrete_f0 + ) + + dataset_size = len(dataset) + + f0_decoder = None + if raw_args.f0_discretization_bounds: + assert task.cfg.discrete_f0 + f0_decoder = Naive_F0_Decoder(raw_args.f0_discretization_bounds).cuda() + + pbar = ( + tqdm.tqdm( + total=dataset_size + if raw_args.max_samples is None + else min(raw_args.max_samples, dataset_size) + ) + if world_size == 1 + else None + ) + + samples_produced = 0 + + for batch in dataloader: + if ( + raw_args.max_samples is not None + and samples_produced >= raw_args.max_samples + ): + break + + prefix = batch["prefix"][0] + + batch = explode_batch(batch, raw_args.batch_explosion_rate) + batch = move_to_cuda(batch) + + if not raw_args.short_curcuit: + produced_tokens, produced_durations, produced_f0, _ = do_sampling( + models[0], + batch, + tgt_dict.eos(), + decoder, + autoregressive_steps=raw_args.max_length - prefix + max_shift, + teacher_force_tokens=raw_args.teacher_force_tokens, + match_duration=raw_args.match_duration, + teacher_force_duration=raw_args.teacher_force_duration, + teacher_force_f0=raw_args.teacher_force_f0, + ) + + # stip entries corresponding to + produced_tokens = produced_tokens[:, 1:] + produced_durations = produced_durations[:, 1:] + produced_f0 = produced_f0[:, 1:] + + else: + max_length = raw_args.max_length + max_shift + produced_tokens, produced_durations, produced_f0 = ( + batch["target"][:, :max_length], + batch["dur_target"][:, :max_length], + batch["f0_target"][:, :max_length], + ) + + if f0_decoder is not None: + produced_f0 = f0_decoder(produced_f0) + + produced_tokens, produced_durations, produced_f0 = ( + produced_tokens.cpu().tolist(), + produced_durations.cpu().tolist(), + produced_f0.cpu().tolist(), + ) + + bsz = batch["target"].size(0) + assert bsz == raw_args.batch_explosion_rate + + for i in range(bsz): + if ( + raw_args.max_samples is not None + and samples_produced >= raw_args.max_samples + ): + break + + produced_tokens_i = produced_tokens[i] + produced_durations_i = produced_durations[i] + produced_f0_i = produced_f0[i] + + ( + produced_tokens_i, + produced_durations_i, + produced_f0_i, + ) = realign_shifted_streams( + produced_tokens_i, produced_durations_i, produced_f0_i, shifts + ) + + produced_tokens_i, produced_durations_i, produced_f0_i = maybe_cut_eos( + produced_tokens_i, produced_durations_i, produced_f0_i, tgt_dict.eos() + ) + + produced_tokens_i, produced_durations_i, produced_f0_i = maybe_filter_pad( + produced_tokens_i, produced_durations_i, produced_f0_i, tgt_dict.pad() + ) + + if raw_args.match_duration: + # NB: here we cheat a bit and use that padding has duration 0 + # so no need to re-align and remove padding + dur_target_i = batch["dur_target"][i, :].sum().item() + produced_tokens_i, produced_durations_i, produced_f0_i = match_duration( + produced_tokens_i, produced_durations_i, produced_f0_i, dur_target_i + ) + + if raw_args.cut_prompt: + produced_tokens_i, produced_durations_i, produced_f0_i = ( + produced_tokens_i[prefix:], + produced_durations_i[prefix:], + produced_f0_i[prefix:], + ) + + prompt_fname = batch["filename"][0] + fname = str(pathlib.Path(prompt_fname).with_suffix("")) + f"__{i}.wav" + + token_stream = unroll_duration(produced_tokens_i, produced_durations_i) + f0_stream = unroll_duration(produced_f0_i, produced_durations_i) + output_line = json.dumps( + { + "audio": fname, + "prompt": prompt_fname, + raw_args.code_type: " ".join(map(str, token_stream)), + "duration": round( + sum(produced_durations_i) + * CODETYPE_TO_FRAMETIME[raw_args.code_type], + 3, + ), + "raw_duration": produced_durations_i, + "raw_f0": produced_f0_i, + "f0": [round(f0, 3) for f0 in f0_stream], + } + ) + print(output_line, file=output_file) + + if pbar: + pbar.update(1) + samples_produced += 1 + + if raw_args.debug: + break + + output_file.close() + + if world_size > 1: + # important that everything is flushed before aggregating + torch.distributed.barrier() + + if world_size > 1 and rank == 0: + with open(raw_args.output, "w") as fout: + for i in range(world_size): + f = raw_args.output + f"_{i}" + with open(f, "r") as fin: + fout.write(fin.read()) + os.remove(f) + + +def cli_main(): + parser = options.get_interactive_generation_parser() + parser.add_argument( + "--prefix-length", + type=int, + default=1, + help="Prompt prefix length (including )", + ) + parser.add_argument("--output", type=str, default=None, required=True) + parser.add_argument( + "--debug", action="store_true", help="Process only the first batch" + ) + parser.add_argument( + "--ignore-durations", + action="store_true", + help="If set, the duration stream is ignored", + ) + parser.add_argument( + "--max-length", type=int, default=200, help="Maximal produced length" + ) + parser.add_argument( + "--code-type", choices=["cpc_km100", "hubert"], default="cpc_km100" + ) + parser.add_argument("--max-samples", type=int, default=None) + parser.add_argument("--prompt-duration-scaler", type=float, default=1.0) + parser.add_argument("--teacher-force-tokens", action="store_true", default=False) + parser.add_argument("--teacher-force-duration", action="store_true", default=False) + parser.add_argument("--teacher-force-f0", action="store_true", default=False) + parser.add_argument("--filter-names", type=str, default=None) + parser.add_argument( + "--match-duration", + action="store_true", + help="Do not produce sequences longer that ground-truth", + ) + parser.add_argument( + "--cut-prompt", + action="store_true", + help="Remove prompt from the produced audio", + ) + parser.add_argument( + "--short-curcuit", action="store_true", help="Use 'target' as a sample" + ) + parser.add_argument("--f0-discretization-bounds", type=str, default=None) + + parser.add_argument("--batch-explosion-rate", type=int, default=1) + + parser.add_argument("--T-token", type=float, default=1.0) + parser.add_argument("--T-duration", type=float, default=1.0) + parser.add_argument("--T-f0", type=float, default=1.0) + + parser.add_argument( + "--subset", type=str, default="valid", choices=["test", "valid"] + ) + + args = options.parse_args_and_arch(parser) + + assert ( + args.prefix_length >= 1 + ), "Prefix length includes bos token , hence the minimum is 1." + assert all( + t >= 0 for t in [args.T_token, args.T_f0, args.T_duration] + ), "T must be non-negative!" + + world_size = torch.cuda.device_count() + if world_size > 1: + import random + + mp.set_start_method("spawn", force=True) + os.environ["MASTER_ADDR"] = "localhost" + os.environ["MASTER_PORT"] = str(random.randint(10_000, 50_000)) + + print(f"Using {world_size} devices, master port {os.environ['MASTER_PORT']}") + + mp.spawn( + main, + nprocs=world_size, + args=( + world_size, + args, + ), + join=True, + ) + else: + main(rank=0, world_size=world_size, args=args) + + +if __name__ == "__main__": + cli_main() diff --git a/data/fairseq/examples/textless_nlp/pgslm/scripts/join_units_manifest.py b/data/fairseq/examples/textless_nlp/pgslm/scripts/join_units_manifest.py new file mode 100644 index 0000000000000000000000000000000000000000..ed14fc5f590bb375336ea6fc7e81094e010254bd --- /dev/null +++ b/data/fairseq/examples/textless_nlp/pgslm/scripts/join_units_manifest.py @@ -0,0 +1,48 @@ +# Copyright (c) Facebook, Inc. and its affiliates. + +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import json +import argparse +import pathlib + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--manifest", required=True) + parser.add_argument("--units", required=True) + parser.add_argument("--output", required=True) + parser.add_argument("--sample_rate", type=int, default=16_000) + + args = parser.parse_args() + + with open(args.manifest, "r") as manifest, open(args.units, "r") as units, open( + args.output, "w" + ) as outp: + root = manifest.readline().strip() + root = pathlib.Path(root) + + for manifest_line, unit_line in zip(manifest.readlines(), units.readlines()): + path, frames = manifest_line.split() + duration = int(frames) / float(args.sample_rate) + fname = root / path + speaker = fname.parent.parent.name + + units = unit_line.split("|")[1] + + print( + json.dumps( + dict( + audio=str(root / path), + duration=duration, + hubert_km100=units.strip(), + speaker=speaker, + ) + ), + file=outp, + ) + + +if __name__ == "__main__": + main() diff --git a/data/fairseq/examples/textless_nlp/pgslm/scripts/prepare_data.sh b/data/fairseq/examples/textless_nlp/pgslm/scripts/prepare_data.sh new file mode 100644 index 0000000000000000000000000000000000000000..ec892e59a4c6aa416e7871e89f443cd046294980 --- /dev/null +++ b/data/fairseq/examples/textless_nlp/pgslm/scripts/prepare_data.sh @@ -0,0 +1,57 @@ +#!/bin/bash +# Copyright (c) Facebook, Inc. and its affiliates. + +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +set -eu + +train_json=$1 +valid_json=$2 +test_json=$3 +n_units=$4 +hop_size=$5 +sr=$6 +f0_quantizer=$7 +out_dir=$8 + +meta_path="$out_dir/data_config.json" +f0_dir="$out_dir/f0" + +mkdir -p $out_dir +ln -sf $train_json $out_dir/train.txt +ln -sf $valid_json $out_dir/valid.txt +ln -sf $test_json $out_dir/test.txt + +cat <$meta_path +{ + "manifests": { + "train": "$out_dir/train.txt", + "valid": "$out_dir/valid.txt", + "test": "$out_dir/test.txt" + }, + "n_units": $n_units, + "code_hop_size": $hop_size, + "sampling_rate": $sr, + "multispkr": "parent_parent_name", + + "f0_vq_type": "naive", + "f0_vq_naive_quantizer": { + "log_mean_norm": "$f0_quantizer" + }, + "f0_vq_n_units": 32 +} +EOF + +for split in train valid test; do + python examples/textless_nlp/pgslm/preprocess_f0.py \ + $out_dir/$split.txt $f0_dir/$split --nshards=1 --rank=1 --sampling_rate=$sr + + #NSHARDS=16 + #seq 1 $NSHARDS | parallel -j $NSHARDS python examples/textless_nlp/pgslm/preprocess_f0.py \ + # $out_dir/$split.txt $f0_dir/$split --nshards=$NSHARDS --sampling_rate=$sr --rank +done + +# Please make sure that the number of shards (--nshards_list) is consistent across commands +python examples/textless_nlp/pgslm/prepare_dataset.py \ + $meta_path $f0_dir --splits test valid train --nshards_list 1 diff --git a/data/fairseq/examples/textless_nlp/pgslm/scripts/prepare_f0_quantization.sh b/data/fairseq/examples/textless_nlp/pgslm/scripts/prepare_f0_quantization.sh new file mode 100644 index 0000000000000000000000000000000000000000..3a285a39bc2bea3240b0f0e38ee71fa872fcadf5 --- /dev/null +++ b/data/fairseq/examples/textless_nlp/pgslm/scripts/prepare_f0_quantization.sh @@ -0,0 +1,27 @@ +#!/bin/bash +# Copyright (c) Facebook, Inc. and its affiliates. + +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +set -eu + +train_json=$1 +sr=$2 +nbins=$3 +out_dir=$4 +out_prefix=$5 + +f0_dir="$out_dir/f0" + +python examples/textless_nlp/pgslm/preprocess_f0.py \ + $train_json $f0_dir/${out_prefix}_f0_quant --nshards 1 --rank 1 --sampling_rate $sr + +# NB: one can use parallel here: +# NSHARDS=16 +# +#seq 1 $NSHARDS | parallel -j $NSHARDS python examples/textless_nlp/pgslm/preprocess_f0.py \ +# $train_json $f0_dir/${out_prefix}_f0_quant --nshards $NSHARDS --sampling_rate $sr --rank + +python examples/textless_nlp/pgslm/quantize_f0.py \ + $train_json $f0_dir/${out_prefix}_f0_quant $out_dir $out_prefix --nbins $nbins --nshards 1 --normalize mean --log diff --git a/data/fairseq/examples/textless_nlp/pgslm/truncated_laplace.py b/data/fairseq/examples/textless_nlp/pgslm/truncated_laplace.py new file mode 100644 index 0000000000000000000000000000000000000000..089f8a8cfce3cbc56ca5d3a7c9bc368184ba780d --- /dev/null +++ b/data/fairseq/examples/textless_nlp/pgslm/truncated_laplace.py @@ -0,0 +1,29 @@ +# Copyright (c) Facebook, Inc. and its affiliates. + +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import torch +import warnings + + +def truncated_laplace(mean, T, truncate_by_zero=False): + """Generating a sample from a Laplace distribution, possible left-truncated at zero. + A bit of explanation here https://stats.stackexchange.com/a/357598 . + """ + assert isinstance(mean, torch.Tensor) + + if not truncate_by_zero: + percentile = 0.0 + else: + if not (mean >= 0.0).all(): + warnings.warn(f"means are supposed to be non-negative, but got {mean}") + mean = torch.clamp_min(mean, 0.0) + + lower_bound = mean.new_tensor([0.0]) + percentile = 0.5 + 0.5 * torch.sign(lower_bound - mean) * ( + 1.0 - torch.exp(-1.0 / T * torch.abs(mean - lower_bound)) + ) + + p = torch.empty_like(mean).uniform_() * (1.0 - percentile) + percentile + return mean - T * torch.sign(p - 0.5) * torch.log(1 - 2 * torch.abs(p - 0.5)) diff --git a/data/fairseq/examples/textless_nlp/speech-resynth/README.md b/data/fairseq/examples/textless_nlp/speech-resynth/README.md new file mode 100644 index 0000000000000000000000000000000000000000..a099682cdbdb991e2c37f1a00387e0b8fc8f121c --- /dev/null +++ b/data/fairseq/examples/textless_nlp/speech-resynth/README.md @@ -0,0 +1,28 @@ + +# Speech Resynthesis from Discrete Disentangled Self-Supervised Representations +Landing page with usfull resources for the [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://arxiv.org/abs/2104.00355) paper. + +

+ +__Abstract__: We propose using self-supervised discrete representations for the task of speech resynthesis. To generate disentangled representation, we separately extract low-bitrate representations for speech content, prosodic information, and speaker identity. This allows to synthesize speech in a controllable manner. We analyze various state-of-the-art, self-supervised representation learning methods and shed light on the advantages of each method while considering reconstruction quality and disentanglement properties. Specifically, we evaluate the F0 reconstruction, speaker identification performance (for both resynthesis and voice conversion), recordings' intelligibility, and overall quality using subjective human evaluation. Lastly, we demonstrate how these representations can be used for an ultra-lightweight speech codec. Using the obtained representations, we can get to a rate of 365 bits per second while providing better speech quality than the baseline methods. + + +## Quick Links +- [Paper](https://arxiv.org/pdf/2104.00355.pdf) +- [Samples](https://speechbot.github.io/resynthesis/index.html) +- [Code](https://github.com/facebookresearch/speech-resynthesis) + +The codebase for the [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://arxiv.org/abs/2104.00355) paper can be found under the following [repository](https://github.com/facebookresearch/speech-resynthesis). + + +## Citation +``` +@inproceedings{polyak21_interspeech, + author={Adam Polyak and Yossi Adi and Jade Copet and + Eugene Kharitonov and Kushal Lakhotia and + Wei-Ning Hsu and Abdelrahman Mohamed and Emmanuel Dupoux}, + title={{Speech Resynthesis from Discrete Disentangled Self-Supervised Representations}}, + year=2021, + booktitle={Proc. Interspeech 2021}, +} +``` diff --git a/data/fairseq/examples/translation_moe/README.md b/data/fairseq/examples/translation_moe/README.md new file mode 100644 index 0000000000000000000000000000000000000000..2e5c8af617f410f64ca38d29447bd05b6af8c5a8 --- /dev/null +++ b/data/fairseq/examples/translation_moe/README.md @@ -0,0 +1,89 @@ +# Mixture Models for Diverse Machine Translation: Tricks of the Trade (Shen et al., 2019) + +This page includes instructions for reproducing results from the paper [Mixture Models for Diverse Machine Translation: Tricks of the Trade (Shen et al., 2019)](https://arxiv.org/abs/1902.07816). + +## Download data + +First, follow the [instructions to download and preprocess the WMT'17 En-De dataset](../translation#prepare-wmt14en2desh). +Make sure to learn a joint vocabulary by passing the `--joined-dictionary` option to `fairseq-preprocess`. + +## Train a model + +Then we can train a mixture of experts model using the `translation_moe` task. +Use the `--method` flag to choose the MoE variant; we support hard mixtures with a learned or uniform prior (`--method hMoElp` and `hMoEup`, respectively) and soft mixures (`--method sMoElp` and `sMoEup`). +The model is trained with online responsibility assignment and shared parameterization. + +The following command will train a `hMoElp` model with `3` experts: +```bash +fairseq-train --ddp-backend='legacy_ddp' \ + data-bin/wmt17_en_de \ + --max-update 100000 \ + --task translation_moe --user-dir examples/translation_moe/translation_moe_src \ + --method hMoElp --mean-pool-gating-network \ + --num-experts 3 \ + --arch transformer_wmt_en_de --share-all-embeddings \ + --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 \ + --lr-scheduler inverse_sqrt --warmup-init-lr 1e-07 --warmup-updates 4000 \ + --lr 0.0007 \ + --dropout 0.1 --weight-decay 0.0 --criterion cross_entropy \ + --max-tokens 3584 +``` + +## Translate + +Once a model is trained, we can generate translations from different experts using the `--gen-expert` option. +For example, to generate from expert 0: +```bash +fairseq-generate data-bin/wmt17_en_de \ + --path checkpoints/checkpoint_best.pt \ + --beam 1 --remove-bpe \ + --task translation_moe --user-dir examples/translation_moe/translation_moe_src \ + --method hMoElp --mean-pool-gating-network \ + --num-experts 3 \ + --gen-expert 0 +``` + +## Evaluate + +First download a tokenized version of the WMT'14 En-De test set with multiple references: +```bash +wget dl.fbaipublicfiles.com/fairseq/data/wmt14-en-de.extra_refs.tok +``` + +Next apply BPE on the fly and run generation for each expert: +```bash +BPE_CODE=examples/translation/wmt17_en_de/code +for EXPERT in $(seq 0 2); do \ + cat wmt14-en-de.extra_refs.tok \ + | grep ^S | cut -f 2 \ + | fairseq-interactive data-bin/wmt17_en_de \ + --path checkpoints/checkpoint_best.pt \ + --beam 1 \ + --bpe subword_nmt --bpe-codes $BPE_CODE \ + --buffer-size 500 --max-tokens 6000 \ + --task translation_moe --user-dir examples/translation_moe/translation_moe_src \ + --method hMoElp --mean-pool-gating-network \ + --num-experts 3 \ + --gen-expert $EXPERT ; \ +done > wmt14-en-de.extra_refs.tok.gen.3experts +``` + +Finally use `score_moe.py` to compute pairwise BLUE and average oracle BLEU: +```bash +python examples/translation_moe/score.py --sys wmt14-en-de.extra_refs.tok.gen.3experts --ref wmt14-en-de.extra_refs.tok +# pairwise BLEU: 48.26 +# #refs covered: 2.11 +# multi-reference BLEU (leave-one-out): 59.46 +``` +This matches row 3 from Table 7 in the paper. + +## Citation + +```bibtex +@article{shen2019mixture, + title = {Mixture Models for Diverse Machine Translation: Tricks of the Trade}, + author = {Tianxiao Shen and Myle Ott and Michael Auli and Marc'Aurelio Ranzato}, + journal = {International Conference on Machine Learning}, + year = 2019, +} +``` diff --git a/data/fairseq/examples/translation_moe/score.py b/data/fairseq/examples/translation_moe/score.py new file mode 100644 index 0000000000000000000000000000000000000000..e45b2cb62e30827bd58cefda9ca6b76a07b855aa --- /dev/null +++ b/data/fairseq/examples/translation_moe/score.py @@ -0,0 +1,197 @@ +#!/usr/bin/env python3 +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. +""" +Scoring script for computing pairwise BLEU and multi-ref BLEU over a set of +candidate hypotheses. + +See `"Mixture Models for Diverse Machine Translation: Tricks of the Trade" +(Shen et al., 2019) `_. +""" + +import argparse +import random +import sys +from itertools import chain + +import numpy as np +import sacrebleu +from sacrebleu import corpus_bleu as _corpus_bleu + +def main(): + parser = argparse.ArgumentParser(sys.argv[0]) + parser.add_argument( + "--sys", nargs="*", default="", metavar="FILE", help="path to system output" + ) + parser.add_argument("--ref", default="", metavar="FILE", help="path to references") + parser.add_argument( + "--output", + default="", + metavar="FILE", + help="print outputs into a pretty format", + ) + args = parser.parse_args() + + if args.sys: + src, tgt, hypos, log_probs = load_sys(args.sys) + print("pairwise BLEU: %.2f" % pairwise(hypos)) + if args.output: + merge(src, tgt, hypos, log_probs, args.output) + + if args.ref: + _, _, refs = load_ref(args.ref) + if args.sys: + multi_ref(refs, hypos) + else: + intra_ref(refs) + + +def dictolist(d): + a = sorted(d.items(), key=lambda i: i[0]) + return [i[1] for i in a] + + +def load_sys(paths): + src, tgt, hypos, log_probs = {}, {}, {}, {} + for path in paths: + with open(path) as f: + for line in f: + line = line.rstrip() + # S: source + # T: target + # D: detokenized system output + if line.startswith(("S-", "T-", "D-")): + i = int(line[line.find("-") + 1 : line.find("\t")]) + if line.startswith("S-"): + src[i] = line.split("\t")[1] + if line.startswith("T-"): + tgt[i] = line.split("\t")[1] + if line.startswith("D-"): + if i not in hypos: + hypos[i] = [] + log_probs[i] = [] + hypos[i].append(line.split("\t")[2]) + log_probs[i].append(float(line.split("\t")[1])) + return dictolist(src), dictolist(tgt), dictolist(hypos), dictolist(log_probs) + + +def load_ref(path): + with open(path) as f: + lines = f.readlines() + src, tgt, refs = [], [], [] + i = 0 + while i < len(lines): + if lines[i].startswith("S-"): + src.append(lines[i].split("\t")[1].rstrip()) + i += 1 + elif lines[i].startswith("T-"): + tgt.append(lines[i].split("\t")[1].rstrip()) + i += 1 + else: + a = [] + while i < len(lines) and lines[i].startswith("R"): + a.append(lines[i].split("\t")[1].rstrip()) + i += 1 + refs.append(a) + return src, tgt, refs + + +def merge(src, tgt, hypos, log_probs, path): + with open(path, "w") as f: + for s, t, hs, lps in zip(src, tgt, hypos, log_probs): + f.write(s + "\n") + f.write(t + "\n") + f.write("\n") + for h, lp in zip(hs, lps): + f.write("\t%f\t%s\n" % (lp, h.strip())) + f.write("------------------------------------------------------\n") + + +def corpus_bleu(sys_stream, ref_streams): + bleu = _corpus_bleu(sys_stream, ref_streams, tokenize="none") + return bleu.score + + +def sentence_bleu(hypothesis, reference): + bleu = _corpus_bleu(hypothesis, reference) + for i in range(1, 4): + bleu.counts[i] += 1 + bleu.totals[i] += 1 + bleu = sacrebleu.BLEU.compute_bleu( + bleu.counts, + bleu.totals, + bleu.sys_len, + bleu.ref_len, + smooth_method="exp", + ) + return bleu.score + + +def pairwise(sents): + _ref, _hypo = [], [] + for s in sents: + for i in range(len(s)): + for j in range(len(s)): + if i != j: + _ref.append(s[i]) + _hypo.append(s[j]) + return corpus_bleu(_hypo, [_ref]) + + +def multi_ref(refs, hypos): + _ref, _hypo = [], [] + ref_cnt = 0 + assert len(refs) == len(hypos) + + # count number of refs covered + for rs, hs in zip(refs, hypos): + a = set() + for h in hs: + s = [sentence_bleu(h, r) for r in rs] + j = np.argmax(s) + _ref.append(rs[j]) + _hypo.append(h) + best = [k for k in range(len(rs)) if s[k] == s[j]] + a.add(random.choice(best)) + ref_cnt += len(a) + print("#refs covered: %.2f" % (ref_cnt / len(refs))) + + # transpose refs and hypos + refs = list(zip(*refs)) + hypos = list(zip(*hypos)) + + # compute multi-ref corpus BLEU (leave-one-out to be comparable to intra_ref) + k = len(hypos) + m = len(refs) + flat_hypos = [hypos[j][i] for i in range(len(hypos[0])) for j in range(k)] + duplicated_refs = [[ref for ref in refs_i for _ in range(k)] for refs_i in refs] + loo_bleus = [] + for held_out_ref in range(m): + remaining_refs = ( + duplicated_refs[:held_out_ref] + duplicated_refs[held_out_ref + 1 :] + ) + assert len(remaining_refs) == m - 1 + loo_bleus.append(corpus_bleu(flat_hypos, remaining_refs)) + print("average multi-reference BLEU (leave-one-out): %.2f" % np.mean(loo_bleus)) + + +def intra_ref(refs): + print("ref pairwise BLEU: %.2f" % pairwise(refs)) + refs = list(zip(*refs)) + m = len(refs) + concat_h = [] + concat_rest = [[] for j in range(m - 1)] + for i, h in enumerate(refs): + rest = refs[:i] + refs[i + 1 :] + concat_h.append(h) + for j in range(m - 1): + concat_rest[j].extend(rest[j]) + concat_h = list(chain.from_iterable(concat_h)) + bleu = corpus_bleu(concat_h, concat_rest) + print("multi-reference BLEU (leave-one-out): %.2f" % bleu) + + +if __name__ == "__main__": + main() diff --git a/data/fairseq/examples/translation_moe/translation_moe_src/__init__.py b/data/fairseq/examples/translation_moe/translation_moe_src/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..c0abe53e973b4bb31cfb062708965d002c79b6e7 --- /dev/null +++ b/data/fairseq/examples/translation_moe/translation_moe_src/__init__.py @@ -0,0 +1,6 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +from . import translation_moe # noqa diff --git a/data/fairseq/examples/translation_moe/translation_moe_src/logsumexp_moe.py b/data/fairseq/examples/translation_moe/translation_moe_src/logsumexp_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..fb299daecbc2b15fb66555bbfb8d1d983e481518 --- /dev/null +++ b/data/fairseq/examples/translation_moe/translation_moe_src/logsumexp_moe.py @@ -0,0 +1,26 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import torch + + +class LogSumExpMoE(torch.autograd.Function): + """Standard LogSumExp forward pass, but use *posterior* for the backward. + + See `"Mixture Models for Diverse Machine Translation: Tricks of the Trade" + (Shen et al., 2019) `_. + """ + + @staticmethod + def forward(ctx, logp, posterior, dim=-1): + ctx.save_for_backward(posterior) + ctx.dim = dim + return torch.logsumexp(logp, dim=dim) + + @staticmethod + def backward(ctx, grad_output): + (posterior,) = ctx.saved_tensors + grad_logp = grad_output.unsqueeze(ctx.dim) * posterior + return grad_logp, None, None diff --git a/data/fairseq/examples/translation_moe/translation_moe_src/mean_pool_gating_network.py b/data/fairseq/examples/translation_moe/translation_moe_src/mean_pool_gating_network.py new file mode 100644 index 0000000000000000000000000000000000000000..efc7ae40bf8fed6c2384cbc6f94477c4caa4c10c --- /dev/null +++ b/data/fairseq/examples/translation_moe/translation_moe_src/mean_pool_gating_network.py @@ -0,0 +1,50 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import torch +import torch.nn.functional as F + + +class MeanPoolGatingNetwork(torch.nn.Module): + """A simple mean-pooling gating network for selecting experts. + + This module applies mean pooling over an encoder's output and returns + reponsibilities for each expert. The encoder format is expected to match + :class:`fairseq.models.transformer.TransformerEncoder`. + """ + + def __init__(self, embed_dim, num_experts, dropout=None): + super().__init__() + self.embed_dim = embed_dim + self.num_experts = num_experts + + self.fc1 = torch.nn.Linear(embed_dim, embed_dim) + self.dropout = torch.nn.Dropout(dropout) if dropout is not None else None + self.fc2 = torch.nn.Linear(embed_dim, num_experts) + + def forward(self, encoder_out): + if not ( + "encoder_out" in encoder_out + and "encoder_padding_mask" in encoder_out + and encoder_out["encoder_out"][0].size(2) == self.embed_dim + ): + raise ValueError("Unexpected format for encoder_out") + + # mean pooling over time + encoder_padding_mask = encoder_out["encoder_padding_mask"][0] # B x T + encoder_out = encoder_out["encoder_out"][0].transpose(0, 1) # B x T x C + if encoder_padding_mask is not None: + encoder_out = encoder_out.clone() # required because of transpose above + encoder_out[encoder_padding_mask] = 0 + ntokens = torch.sum(~encoder_padding_mask, dim=1, keepdim=True) + x = torch.sum(encoder_out, dim=1) / ntokens.type_as(encoder_out) + else: + x = torch.mean(encoder_out, dim=1) + + x = torch.tanh(self.fc1(x)) + if self.dropout is not None: + x = self.dropout(x) + x = self.fc2(x) + return F.log_softmax(x, dim=-1, dtype=torch.float32).type_as(x) diff --git a/data/fairseq/examples/translation_moe/translation_moe_src/translation_moe.py b/data/fairseq/examples/translation_moe/translation_moe_src/translation_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..a829bf7dcbc79b4fddc2a09d6c57134871aa1944 --- /dev/null +++ b/data/fairseq/examples/translation_moe/translation_moe_src/translation_moe.py @@ -0,0 +1,259 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +from dataclasses import dataclass, field +import torch +from omegaconf import II + +from fairseq import utils +from fairseq.logging import metrics +from fairseq.dataclass import ChoiceEnum +from fairseq.tasks import register_task +from fairseq.tasks.translation import TranslationConfig, TranslationTask + +from .logsumexp_moe import LogSumExpMoE +from .mean_pool_gating_network import MeanPoolGatingNetwork + + +METHOD_CHOICES = ChoiceEnum(["sMoElp", "sMoEup", "hMoElp", "hMoEup"]) + + +@dataclass +class TranslationMoEConfig(TranslationConfig): + method: METHOD_CHOICES = field( + default="hMoEup", + metadata={"help": "MoE method"}, + ) + num_experts: int = field( + default=3, + metadata={"help": "number of experts"}, + ) + mean_pool_gating_network: bool = field( + default=False, + metadata={"help": "use a simple mean-pooling gating network"}, + ) + mean_pool_gating_network_dropout: float = field( + default=0, + metadata={"help": "dropout for mean-pooling gating network"}, + ) + mean_pool_gating_network_encoder_dim: int = field( + default=0, + metadata={"help": "encoder output dim for mean-pooling gating network"}, + ) + gen_expert: int = field( + default=0, + metadata={"help": "which expert to use for generation"}, + ) + sentence_avg: bool = II("optimization.sentence_avg") + + +@register_task("translation_moe", dataclass=TranslationMoEConfig) +class TranslationMoETask(TranslationTask): + """ + Translation task for Mixture of Experts (MoE) models. + + See `"Mixture Models for Diverse Machine Translation: Tricks of the Trade" + (Shen et al., 2019) `_. + + Args: + src_dict (~fairseq.data.Dictionary): dictionary for the source language + tgt_dict (~fairseq.data.Dictionary): dictionary for the target language + + .. note:: + + The translation task is compatible with :mod:`fairseq-train`, + :mod:`fairseq-generate` and :mod:`fairseq-interactive`. + + The translation task provides the following additional command-line + arguments: + + .. argparse:: + :ref: fairseq.tasks.translation_parser + :prog: + """ + + cfg: TranslationMoEConfig + + def __init__(self, cfg: TranslationMoEConfig, src_dict, tgt_dict): + if cfg.method == "sMoElp": + # soft MoE with learned prior + self.uniform_prior = False + self.hard_selection = False + elif cfg.method == "sMoEup": + # soft MoE with uniform prior + self.uniform_prior = True + self.hard_selection = False + elif cfg.method == "hMoElp": + # hard MoE with learned prior + self.uniform_prior = False + self.hard_selection = True + elif cfg.method == "hMoEup": + # hard MoE with uniform prior + self.uniform_prior = True + self.hard_selection = True + + # add indicator tokens for each expert + for i in range(cfg.num_experts): + # add to both dictionaries in case we're sharing embeddings + src_dict.add_symbol("".format(i)) + tgt_dict.add_symbol("".format(i)) + + super().__init__(cfg, src_dict, tgt_dict) + + def build_model(self, cfg, from_checkpoint=False): + from fairseq import models + + model = models.build_model(cfg, self) + if not self.uniform_prior and not hasattr(model, "gating_network"): + if self.cfg.mean_pool_gating_network: + if self.cfg.mean_pool_gating_network_encoder_dim > 0: + encoder_dim = self.cfg.mean_pool_gating_network_encoder_dim + elif getattr(cfg, "encoder_embed_dim", None): + # assume that encoder_embed_dim is the encoder's output dimension + encoder_dim = cfg.encoder_embed_dim + else: + raise ValueError( + "Must specify --mean-pool-gating-network-encoder-dim" + ) + + if self.cfg.mean_pool_gating_network_dropout > 0: + dropout = self.cfg.mean_pool_gating_network_dropout + elif getattr(cfg, "dropout", None): + dropout = cfg.dropout + else: + raise ValueError("Must specify task.mean_pool_gating_network_dropout") + + model.gating_network = MeanPoolGatingNetwork( + encoder_dim, + self.cfg.num_experts, + dropout, + ) + else: + raise ValueError( + "translation_moe task with learned prior requires the model to " + "have a gating network; try using --mean-pool-gating-network" + ) + return model + + def expert_index(self, i): + return i + self.tgt_dict.index("") + + def _get_loss(self, sample, model, criterion): + assert hasattr( + criterion, "compute_loss" + ), "translation_moe task requires the criterion to implement the compute_loss() method" + + k = self.cfg.num_experts + bsz = sample["target"].size(0) + + def get_lprob_y(encoder_out, prev_output_tokens_k): + net_output = model.decoder( + prev_output_tokens=prev_output_tokens_k, + encoder_out=encoder_out, + ) + loss, _ = criterion.compute_loss(model, net_output, sample, reduce=False) + loss = loss.view(bsz, -1) + return -loss.sum(dim=1, keepdim=True) # -> B x 1 + + def get_lprob_yz(winners=None): + encoder_out = model.encoder( + src_tokens=sample["net_input"]["src_tokens"], + src_lengths=sample["net_input"]["src_lengths"], + ) + + if winners is None: + lprob_y = [] + for i in range(k): + prev_output_tokens_k = sample["net_input"][ + "prev_output_tokens" + ].clone() + assert not prev_output_tokens_k.requires_grad + prev_output_tokens_k[:, 0] = self.expert_index(i) + lprob_y.append(get_lprob_y(encoder_out, prev_output_tokens_k)) + lprob_y = torch.cat(lprob_y, dim=1) # -> B x K + else: + prev_output_tokens_k = sample["net_input"]["prev_output_tokens"].clone() + prev_output_tokens_k[:, 0] = self.expert_index(winners) + lprob_y = get_lprob_y(encoder_out, prev_output_tokens_k) # -> B + + if self.uniform_prior: + lprob_yz = lprob_y + else: + lprob_z = model.gating_network(encoder_out) # B x K + if winners is not None: + lprob_z = lprob_z.gather(dim=1, index=winners.unsqueeze(-1)) + lprob_yz = lprob_y + lprob_z.type_as(lprob_y) # B x K + + return lprob_yz + + # compute responsibilities without dropout + with utils.model_eval(model): # disable dropout + with torch.no_grad(): # disable autograd + lprob_yz = get_lprob_yz() # B x K + prob_z_xy = torch.nn.functional.softmax(lprob_yz, dim=1) + assert not prob_z_xy.requires_grad + + # compute loss with dropout + if self.hard_selection: + winners = prob_z_xy.max(dim=1)[1] + loss = -get_lprob_yz(winners) + else: + lprob_yz = get_lprob_yz() # B x K + loss = -LogSumExpMoE.apply(lprob_yz, prob_z_xy, 1) + + loss = loss.sum() + sample_size = ( + sample["target"].size(0) if self.cfg.sentence_avg else sample["ntokens"] + ) + logging_output = { + "loss": utils.item(loss.data), + "ntokens": sample["ntokens"], + "nsentences": bsz, + "sample_size": sample_size, + "posterior": prob_z_xy.float().sum(dim=0).cpu(), + } + return loss, sample_size, logging_output + + def train_step( + self, sample, model, criterion, optimizer, update_num, ignore_grad=False + ): + model.train() + loss, sample_size, logging_output = self._get_loss(sample, model, criterion) + if ignore_grad: + loss *= 0 + optimizer.backward(loss) + return loss, sample_size, logging_output + + def valid_step(self, sample, model, criterion): + model.eval() + with torch.no_grad(): + loss, sample_size, logging_output = self._get_loss(sample, model, criterion) + return loss, sample_size, logging_output + + def inference_step( + self, + generator, + models, + sample, + prefix_tokens=None, + expert=None, + constraints=None, + ): + expert = expert or self.cfg.gen_expert + with torch.no_grad(): + return generator.generate( + models, + sample, + prefix_tokens=prefix_tokens, + constraints=constraints, + bos_token=self.expert_index(expert), + ) + + def reduce_metrics(self, logging_outputs, criterion): + super().reduce_metrics(logging_outputs, criterion) + metrics.log_scalar( + "posterior", + sum(log["posterior"] for log in logging_outputs if "posterior" in log), + ) diff --git a/data/fairseq/examples/unsupervised_quality_estimation/README.md b/data/fairseq/examples/unsupervised_quality_estimation/README.md new file mode 100644 index 0000000000000000000000000000000000000000..e86a0d13b883af0c37fdc2c1fee9b0b9dff4d18c --- /dev/null +++ b/data/fairseq/examples/unsupervised_quality_estimation/README.md @@ -0,0 +1,126 @@ +# Unsupervised Quality Estimation for Neural Machine Translation (Fomicheva et al., 2020) + +This page includes instructions for reproducing results from the paper [Unsupervised Quality Estimation for Neural +Machine Translation (Fomicheva et al., 2020)](https://arxiv.org/abs/2005.10608) + +## Requirements: + +* mosesdecoder: https://github.com/moses-smt/mosesdecoder +* subword-nmt: https://github.com/rsennrich/subword-nmt +* flores: https://github.com/facebookresearch/flores + +## Download Models and Test Data + +Download translation models and test data from [MLQE dataset repository](https://github.com/facebookresearch/mlqe). + +## Set up: + +Given a testset consisting of source sentences and reference translations: + +* `SRC_LANG`: source language +* `TGT_LANG`: target language +* `INPUT`: input prefix, such that the file `$INPUT.$SRC_LANG` contains source sentences and `$INPUT.$TGT_LANG` +contains the reference sentences +* `OUTPUT_DIR`: output path to store results +* `MOSES_DECODER`: path to mosesdecoder installation +* `BPE_ROOT`: path to subword-nmt installation +* `BPE`: path to BPE model +* `MODEL_DIR`: directory containing the NMT model `.pt` file as well as the source and target vocabularies. +* `TMP`: directory for intermediate temporary files +* `GPU`: if translating with GPU, id of the GPU to use for inference +* `DROPOUT_N`: number of stochastic forward passes + +`$DROPOUT_N` is set to 30 in the experiments reported in the paper. However, we observed that increasing it beyond 10 +does not bring substantial improvements. + +## Translate the data using standard decoding + +Preprocess the input data: +``` +for LANG in $SRC_LANG $TGT_LANG; do + perl $MOSES_DECODER/scripts/tokenizer/tokenizer.perl -threads 80 -a -l $LANG < $INPUT.$LANG > $TMP/preprocessed.tok.$LANG + python $BPE_ROOT/apply_bpe.py -c ${BPE} < $TMP/preprocessed.tok.$LANG > $TMP/preprocessed.tok.bpe.$LANG +done +``` + +Binarize the data for faster translation: + +``` +fairseq-preprocess --srcdict $MODEL_DIR/dict.$SRC_LANG.txt --tgtdict $MODEL_DIR/dict.$TGT_LANG.txt +--source-lang ${SRC_LANG} --target-lang ${TGT_LANG} --testpref $TMP/preprocessed.tok.bpe --destdir $TMP/bin --workers 4 +``` + +Translate + +``` +CUDA_VISIBLE_DEVICES=$GPU fairseq-generate $TMP/bin --path ${MODEL_DIR}/${SRC_LANG}-${TGT_LANG}.pt --beam 5 +--source-lang $SRC_LANG --target-lang $TGT_LANG --no-progress-bar --unkpen 5 > $TMP/fairseq.out +grep ^H $TMP/fairseq.out | cut -d- -f2- | sort -n | cut -f3- > $TMP/mt.out +``` + +Post-process + +``` +sed -r 's/(@@ )| (@@ ?$)//g' < $TMP/mt.out | perl $MOSES_DECODER/scripts/tokenizer/detokenizer.perl +-l $TGT_LANG > $OUTPUT_DIR/mt.out +``` + +## Produce uncertainty estimates + +### Scoring + +Make temporary files to store the translations repeated N times. + +``` +python ${SCRIPTS}/scripts/uncertainty/repeat_lines.py -i $TMP/preprocessed.tok.bpe.$SRC_LANG -n $DROPOUT_N +-o $TMP/repeated.$SRC_LANG +python ${SCRIPTS}/scripts/uncertainty/repeat_lines.py -i $TMP/mt.out -n $DROPOUT_N -o $TMP/repeated.$TGT_LANG + +fairseq-preprocess --srcdict ${MODEL_DIR}/dict.${SRC_LANG}.txt $TGT_DIC --source-lang ${SRC_LANG} +--target-lang ${TGT_LANG} --testpref ${TMP}/repeated --destdir ${TMP}/bin-repeated +``` + +Produce model scores for the generated translations using `--retain-dropout` option to apply dropout at inference time: + +``` +CUDA_VISIBLE_DEVICES=${GPU} fairseq-generate ${TMP}/bin-repeated --path ${MODEL_DIR}/${LP}.pt --beam 5 + --source-lang $SRC_LANG --target-lang $TGT_LANG --no-progress-bar --unkpen 5 --score-reference --retain-dropout + --retain-dropout-modules '["TransformerModel","TransformerEncoder","TransformerDecoder","TransformerEncoderLayer"]' + TransformerDecoderLayer --seed 46 > $TMP/dropout.scoring.out + +grep ^H $TMP/dropout.scoring.out | cut -d- -f2- | sort -n | cut -f2 > $TMP/dropout.scores + +``` + +Use `--retain-dropout-modules` to specify the modules. By default, dropout is applied in the same places +as for training. + +Compute the mean of the resulting output distribution: + +``` +python $SCRIPTS/scripts/uncertainty/aggregate_scores.py -i $TMP/dropout.scores -o $OUTPUT_DIR/dropout.scores.mean +-n $DROPOUT_N +``` + +### Generation + +Produce multiple translation hypotheses for the same source using `--retain-dropout` option: + +``` +CUDA_VISIBLE_DEVICES=${GPU} fairseq-generate ${TMP}/bin-repeated --path ${MODEL_DIR}/${LP}.pt + --beam 5 --source-lang $SRC_LANG --target-lang $TGT_LANG --no-progress-bar --retain-dropout + --unkpen 5 --retain-dropout-modules TransformerModel TransformerEncoder TransformerDecoder +TransformerEncoderLayer TransformerDecoderLayer --seed 46 > $TMP/dropout.generation.out + +grep ^H $TMP/dropout.generation.out | cut -d- -f2- | sort -n | cut -f3- > $TMP/dropout.hypotheses_ + +sed -r 's/(@@ )| (@@ ?$)//g' < $TMP/dropout.hypotheses_ | perl $MOSES_DECODER/scripts/tokenizer/detokenizer.perl +-l $TGT_LANG > $TMP/dropout.hypotheses +``` + +Compute similarity between multiple hypotheses corresponding to the same source sentence using Meteor +evaluation metric: +``` +python meteor.py -i $TMP/dropout.hypotheses -m -n $DROPOUT_N -o +$OUTPUT_DIR/dropout.gen.sim.meteor +``` diff --git a/data/fairseq/examples/unsupervised_quality_estimation/aggregate_scores.py b/data/fairseq/examples/unsupervised_quality_estimation/aggregate_scores.py new file mode 100644 index 0000000000000000000000000000000000000000..66d50d07ff2067b802b90a2aadd88df23153830a --- /dev/null +++ b/data/fairseq/examples/unsupervised_quality_estimation/aggregate_scores.py @@ -0,0 +1,41 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import argparse +import sys + +import numpy as np + + +aggregate_funcs = { + "std": np.std, + "var": np.var, + "median": np.median, + "mean": np.mean, + "min": np.min, + "max": np.max, +} + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("-i", "--input_file", required=True, type=str) + parser.add_argument("-n", "--repeat_times", required=True, type=int) + parser.add_argument("-o", "--output_file", required=False) + parser.add_argument("-f", "--func", required=False, default="mean") + args = parser.parse_args() + + stream = open(args.output_file, "w") if args.output_file else sys.stdout + + segment_scores = [] + for line in open(args.input_file): + segment_scores.append(float(line.strip())) + if len(segment_scores) == args.repeat_times: + stream.write("{}\n".format(aggregate_funcs[args.func](segment_scores))) + segment_scores = [] + + +if __name__ == "__main__": + main() diff --git a/data/fairseq/examples/unsupervised_quality_estimation/meteor.py b/data/fairseq/examples/unsupervised_quality_estimation/meteor.py new file mode 100644 index 0000000000000000000000000000000000000000..2ee0448cf1f167f6f3ecee56ad807922cffb0956 --- /dev/null +++ b/data/fairseq/examples/unsupervised_quality_estimation/meteor.py @@ -0,0 +1,109 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import argparse +import math +import os +import subprocess +import sys +import tempfile +from collections import defaultdict +from itertools import combinations + + +def read_translations(path, n_repeats): + segment_counter = 0 + segment_translations = [] + translations = defaultdict(list) + for line in open(path): + segment_translations.append(" ".join(line.split())) + if len(segment_translations) == n_repeats: + translations[segment_counter] = segment_translations + segment_translations = [] + segment_counter += 1 + return translations + + +def generate_input(translations, n_repeats): + _, ref_path = tempfile.mkstemp() + _, mt_path = tempfile.mkstemp() + ref_fh = open(ref_path, "w") + mt_fh = open(mt_path, "w") + for segid in sorted(translations.keys()): + assert len(translations[segid]) == n_repeats + indexes = combinations(range(n_repeats), 2) + for idx1, idx2 in indexes: + mt_fh.write(translations[segid][idx1].strip() + "\n") + ref_fh.write(translations[segid][idx2].strip() + "\n") + sys.stderr.write("\nSaved translations to %s and %s" % (ref_path, mt_path)) + return ref_path, mt_path + + +def run_meteor(ref_path, mt_path, metric_path, lang="en"): + _, out_path = tempfile.mkstemp() + subprocess.call( + [ + "java", + "-Xmx2G", + "-jar", + metric_path, + mt_path, + ref_path, + "-p", + "0.5 0.2 0.6 0.75", # default parameters, only changed alpha to give equal weight to P and R + "-norm", + "-l", + lang, + ], + stdout=open(out_path, "w"), + ) + os.remove(ref_path) + os.remove(mt_path) + sys.stderr.write("\nSaved Meteor output to %s" % out_path) + return out_path + + +def read_output(meteor_output_path, n_repeats): + n_combinations = math.factorial(n_repeats) / ( + math.factorial(2) * math.factorial(n_repeats - 2) + ) + raw_scores = [] + average_scores = [] + for line in open(meteor_output_path): + if not line.startswith("Segment "): + continue + score = float(line.strip().split("\t")[1]) + raw_scores.append(score) + if len(raw_scores) == n_combinations: + average_scores.append(sum(raw_scores) / n_combinations) + raw_scores = [] + os.remove(meteor_output_path) + return average_scores + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("-i", "--infile") + parser.add_argument("-n", "--repeat_times", type=int) + parser.add_argument("-m", "--meteor") + parser.add_argument("-o", "--output") + args = parser.parse_args() + + translations = read_translations(args.infile, args.repeat_times) + sys.stderr.write("\nGenerating input for Meteor...") + ref_path, mt_path = generate_input(translations, args.repeat_times) + sys.stderr.write("\nRunning Meteor...") + out_path = run_meteor(ref_path, mt_path, args.meteor) + sys.stderr.write("\nReading output...") + scores = read_output(out_path, args.repeat_times) + sys.stderr.write("\nWriting results...") + with open(args.output, "w") as o: + for scr in scores: + o.write("{}\n".format(scr)) + o.close() + + +if __name__ == "__main__": + main() diff --git a/data/fairseq/examples/unsupervised_quality_estimation/repeat_lines.py b/data/fairseq/examples/unsupervised_quality_estimation/repeat_lines.py new file mode 100644 index 0000000000000000000000000000000000000000..5a04851a74624e9c8ebc259805b7aed6c638b0de --- /dev/null +++ b/data/fairseq/examples/unsupervised_quality_estimation/repeat_lines.py @@ -0,0 +1,28 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import argparse +import sys + + +def _normalize_spaces(line): + return " ".join(line.split()) + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("-i", "--input_file", required=True, type=str) + parser.add_argument("-n", "--repeat_times", required=True, type=int) + parser.add_argument("-o", "--output_file", required=False, type=str) + args = parser.parse_args() + stream = open(args.output_file, "w") if args.output_file else sys.stdout + + for line in open(args.input_file): + for _ in range(args.repeat_times): + stream.write(_normalize_spaces(line) + "\n") + + +if __name__ == "__main__": + main() diff --git a/data/fairseq/examples/wmt21/README.md b/data/fairseq/examples/wmt21/README.md new file mode 100644 index 0000000000000000000000000000000000000000..524fffb7247bce6ab2f54e2124bcc3d680702778 --- /dev/null +++ b/data/fairseq/examples/wmt21/README.md @@ -0,0 +1,25 @@ +# WMT 21 + +This page provides pointers to the models of Facebook AI's WMT'21 news translation task submission [(Tran et al., 2021)](https://arxiv.org/abs/2108.03265). + +## Single best dense models + +Model | Description | Download +---|---|--- +`wmt21.dense-24-wide.X-En` | X-En | [download (.tar.gz)](https://dl.fbaipublicfiles.com/fairseq/models/wmt21.dense-24-wide.X-En.tar.gz) +`wmt21.dense-24-wide.En-X` | En-X | [download (.tar.gz)](https://dl.fbaipublicfiles.com/fairseq/models/wmt21.dense-24-wide.En-X.tar.gz) + +## Example usage + +See eval.sh + + +## Citation +```bibtex +@inproceedings{tran2021facebook + title={Facebook AI’s WMT21 News Translation Task Submission}, + author={Chau Tran and Shruti Bhosale and James Cross and Philipp Koehn and Sergey Edunov and Angela Fan}, + booktitle={Proc. of WMT}, + year={2021}, +} +``` diff --git a/data/fairseq/examples/wmt21/eval.sh b/data/fairseq/examples/wmt21/eval.sh new file mode 100644 index 0000000000000000000000000000000000000000..b36d934c51b948a54f11a62cbc486a17431eea89 --- /dev/null +++ b/data/fairseq/examples/wmt21/eval.sh @@ -0,0 +1,49 @@ +#!/bin/bash +# Copyright (c) Facebook, Inc. and its affiliates. +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. +SRC=en +TGT=is +MODEL_NAME=wmt21.dense-24-wide.En-X + +PATH_TO_FAIRSEQ_PY=. +TMP_DIR=generation_tmp +mkdir -p $TMP_DIR + +REPLACE_UNICODE_PUNCT=$PATH_TO_FAIRSEQ_PY/examples/wmt21/scripts/replace-unicode-punctuation.perl +NORM_PUNCT=$PATH_TO_FAIRSEQ_PY/examples/wmt21/scripts/normalize-punctuation.perl +if [ ! -d "${TMP_DIR}/${MODEL_NAME}" ]; then + wget https://dl.fbaipublicfiles.com/fairseq/models/${MODEL_NAME}.tar.gz -P $TMP_DIR/ + tar -xvf $TMP_DIR/${MODEL_NAME}.tar.gz -C $TMP_DIR +fi +MODEL_DIR=$TMP_DIR/${MODEL_NAME} +if [ ! -d "${TMP_DIR}/wmt21-news-systems" ]; then + git clone https://github.com/wmt-conference/wmt21-news-systems $TMP_DIR/wmt21-news-systems +fi + +DOMAIN_TAG="wmtdata newsdomain" +INPUT_FILE=$TMP_DIR/wmt21-news-systems/txt/sources/newstest2021.${SRC}-${TGT}.src.${SRC} +REF_FILE=$TMP_DIR/wmt21-news-systems/txt/references/newstest2021.${SRC}-${TGT}.ref.A.${TGT} + +# Translate +cat ${INPUT_FILE} | sed "s/^/${DOMAIN_TAG} /" | $REPLACE_UNICODE_PUNCT | $NORM_PUNCT -l ${SRC} | python $PATH_TO_FAIRSEQ_PY/fairseq_cli/interactive.py $MODEL_DIR \ + --path ${MODEL_DIR}/checkpoint.pt \ + --task translation_multi_simple_epoch \ + --langs "en,ha,is,ja,cs,ru,zh,de" \ + --lang-pairs $SRC-$TGT \ + --bpe "sentencepiece" \ + --sentencepiece-model ${MODEL_DIR}/sentencepiece.model \ + --buffer-size 1024 \ + --batch-size 10 -s $SRC -t $TGT \ + --decoder-langtok \ + --encoder-langtok src \ + --beam 5 \ + --lenpen 1.0 \ + --fp16 > $TMP_DIR/${SRC}-${TGT}.gen_log + +cat $TMP_DIR/$SRC-$TGT.gen_log | grep -P "^D-" | cut -f3 > $TMP_DIR/$SRC-$TGT.hyp + +# Calculate BLEU score +sacrebleu -l $SRC-$TGT $REF_FILE < $TMP_DIR/$SRC-$TGT.hyp diff --git a/data/fairseq/examples/wmt21/scripts/normalize-punctuation.perl b/data/fairseq/examples/wmt21/scripts/normalize-punctuation.perl new file mode 100644 index 0000000000000000000000000000000000000000..a7c0750f5844ab3a10a497714ce55dcf8680cf49 --- /dev/null +++ b/data/fairseq/examples/wmt21/scripts/normalize-punctuation.perl @@ -0,0 +1,90 @@ +#!/usr/bin/env perl +# +# This file is part of moses. Its use is licensed under the GNU Lesser General +# Public License version 2.1 or, at your option, any later version. + +use warnings; +use strict; + +my $language = "en"; +my $PENN = 0; + +while (@ARGV) { + $_ = shift; + /^-b$/ && ($| = 1, next); # not buffered (flush each line) + /^-l$/ && ($language = shift, next); + /^[^\-]/ && ($language = $_, next); + /^-penn$/ && ($PENN = 1, next); +} + +while() { + s/\r//g; + # remove extra spaces + s/\(/ \(/g; + s/\)/\) /g; s/ +/ /g; + s/\) ([\.\!\:\?\;\,])/\)$1/g; + s/\( /\(/g; + s/ \)/\)/g; + s/(\d) \%/$1\%/g; + s/ :/:/g; + s/ ;/;/g; + # normalize unicode punctuation + if ($PENN == 0) { + s/\`/\'/g; + s/\'\'/ \" /g; + } + + s/„/\"/g; + s/“/\"/g; + s/”/\"/g; + s/–/-/g; + s/—/ - /g; s/ +/ /g; + s/´/\'/g; + s/([a-z])‘([a-z])/$1\'$2/gi; + s/([a-z])’([a-z])/$1\'$2/gi; + s/‘/\'/g; + s/‚/\'/g; + s/’/\"/g; + s/''/\"/g; + s/´´/\"/g; + s/…/.../g; + # French quotes + s/ « / \"/g; + s/« /\"/g; + s/«/\"/g; + s/ » /\" /g; + s/ »/\"/g; + s/»/\"/g; + # handle pseudo-spaces + s/ \%/\%/g; + s/nº /nº /g; + s/ :/:/g; + s/ ºC/ ºC/g; + s/ cm/ cm/g; + s/ \?/\?/g; + s/ \!/\!/g; + s/ ;/;/g; + s/, /, /g; s/ +/ /g; + + # English "quotation," followed by comma, style + if ($language eq "en") { + s/\"([,\.]+)/$1\"/g; + } + # Czech is confused + elsif ($language eq "cs" || $language eq "cz") { + } + # German/Spanish/French "quotation", followed by comma, style + else { + s/,\"/\",/g; + s/(\.+)\"(\s*[^<])/\"$1$2/g; # don't fix period at end of sentence + } + + + if ($language eq "de" || $language eq "es" || $language eq "cz" || $language eq "cs" || $language eq "fr") { + s/(\d) (\d)/$1,$2/g; + } + else { + s/(\d) (\d)/$1.$2/g; + } + print $_; +} diff --git a/data/fairseq/examples/wmt21/scripts/replace-unicode-punctuation.perl b/data/fairseq/examples/wmt21/scripts/replace-unicode-punctuation.perl new file mode 100644 index 0000000000000000000000000000000000000000..faed2cd9d86af7ad96300571b3a7b1037cc21e79 --- /dev/null +++ b/data/fairseq/examples/wmt21/scripts/replace-unicode-punctuation.perl @@ -0,0 +1,55 @@ +#!/usr/bin/env perl +# +# This file is part of moses. Its use is licensed under the GNU Lesser General +# Public License version 2.1 or, at your option, any later version. + +use warnings; +use strict; + +while (@ARGV) { + $_ = shift; + /^-b$/ && ($| = 1, next); # not buffered (flush each line) +} + +#binmode(STDIN, ":utf8"); +#binmode(STDOUT, ":utf8"); + +while() { + s/,/,/g; + s/。 */. /g; + s/、/,/g; + s/”/"/g; + s/“/"/g; + s/∶/:/g; + s/:/:/g; + s/?/\?/g; + s/《/"/g; + s/》/"/g; + s/)/\)/g; + s/!/\!/g; + s/(/\(/g; + s/;/;/g; + s/1/1/g; + s/」/"/g; + s/「/"/g; + s/0/0/g; + s/3/3/g; + s/2/2/g; + s/5/5/g; + s/6/6/g; + s/9/9/g; + s/7/7/g; + s/8/8/g; + s/4/4/g; + s/. */. /g; + s/~/\~/g; + s/’/\'/g; + s/…/\.\.\./g; + s/━/\-/g; + s/〈/\/g; + s/【/\[/g; + s/】/\]/g; + s/%/\%/g; + print $_; +} diff --git a/data/fairseq/examples/xglm/README.md b/data/fairseq/examples/xglm/README.md new file mode 100644 index 0000000000000000000000000000000000000000..914e297669a1f92096a39d43f366309d57c52854 --- /dev/null +++ b/data/fairseq/examples/xglm/README.md @@ -0,0 +1,195 @@ +# Few-shot Learning with Multilingual Language Models + +## Introduction + +In this work, we train a family of multilingual generative language models, dubbed XGLM, on a balanced corpus covering a diverse set of languages, and study their few- and zero-shot learning capabilities in a wide range of tasks. Our largest model with 7.5 billion parameters sets new state of the art in few-shot learning on more than 20 representative languages, outperforming GPT-3 of comparable size in multilingual commonsense reasoning (+7.4 accuracy points for 0-shot, +9.4 for 4-shot) and natural language inference (+5.4 for 0-shot, +5.4 for 4-shot). We have included a [model card](model_card.md) of XGLM for transparency and accountability. + +## Data and Languages +XGLM models are trained on a new multilingual corpus extracted from CommonCrawl (CC100-XL), a significantly larger multilingual dataset covering 68 Common Crawl (CC) snapshots (from [Summer 2013](http://commoncrawl.org/2013/11/new-crawl-data-available/) to [March/April 2020](https://commoncrawl.org/2020/04/march-april-2020-crawl-archive-now-available/) consisting of 134 languages. The detailed languages and data statistics are reported in the paper (Table A.1). + +## Pre-trained models + +Model | Layers | Model Dim | FFN Dim | Languages | Download +---|---|---|---|---|--- +`XGLM 564M` | 24 | 1024 | 4096 | trained on 30 languages| [xglm.564M.tar.gz](https://dl.fbaipublicfiles.com/fairseq/models/xglm/xglm.564M.tar.gz) +`XGLM 1.7B` | 24 | 2048 | 8192 | trained on 30 languages| [xglm.1.7B.tar.gz](https://dl.fbaipublicfiles.com/fairseq/models/xglm/xglm.1.7B.tar.gz) +`XGLM 2.9B` | 48 | 2048 | 8192 | trained on 30 languages| [xglm.2.9B.tar.gz](https://dl.fbaipublicfiles.com/fairseq/models/xglm/xglm.2.9B.tar.gz) +`XGLM 7.5B` | 32 | 4096 | 16384 | trained on 30 languages| [xglm.7.5B.tar.gz](https://dl.fbaipublicfiles.com/fairseq/models/xglm/xglm.7.5B.tar.gz) +`XGLM 4.5B` | 48 | 2048 | 16384 | trained on 134 languages| [xglm.4.5B.tar.gz](https://dl.fbaipublicfiles.com/fairseq/models/xglm/xglm.4.5B.tar.gz) + +## Pre-training Data Format +Our models were pre-trained with data in the following format (i.e. paragraphs are separated with new lines and documents were separated with double new lines). +``` + ... # X0: number of tokens in para0 of doc0 + ... # Y0: number of tokens in para1 of doc0 + + ... # X1: number of tokens in para0 of doc1 + ... # Y1: number of tokens in para1 of doc1 + +... +``` +Fairseq's preprocessing replaces newlines with the end-of-sentence symbol (`
`). As a result, the models never saw newline characters during pretraining and the same preprocessing should be run prior to few-shot inference to maximize performance. For example, our language model scoring function has `replace_newlines_with_eos` argument to trigger this preprocessing: +```python +from fairseq.models.transformer_lm import TransformerLanguageModel + +model_dir = 'path_to_decompressed_tar_gz_dir' +lm = TransformerLanguageModel.from_pretrained(model_dir, bpe='sentencepiece') + +text = """First paragraph of the first document. +Second paragraph of the first document. + +First paragraph of the second document. +""" +tokens = lm.score(text, replace_newlines_with_eos=True)['tokens'] +assert '\n' not in lm.decode(tokens) # no newlines were encoded +``` + +## Evaluation + +### Example (COPA) + +The following snippet show how to evaluate our models on the Choice of Plausible Alternatives (COPA) task, using examples in English, Chinese and Hindi. + +```python +data_samples = { + 'en': [ + { + "premise": "I wanted to conserve energy.", + "choice1": "I swept the floor in the unoccupied room.", + "choice2": "I shut off the light in the unoccupied room.", + "question": "effect", + "label": "1" + }, + { + "premise": "The flame on the candle went out.", + "choice1": "I blew on the wick.", + "choice2": "I put a match to the wick.", + "question": "cause", + "label": "0" + } + ], + 'zh': [ + { + "premise": "我想节约能源。", + "choice1": "我在空着的房间里扫了地板。", + "choice2": "我把空房间里的灯关了。", + "question": "effect", + "label": "1" + }, + { + "premise": "蜡烛上的火焰熄灭了。", + "choice1": "我吹灭了灯芯。", + "choice2": "我把一根火柴放在灯芯上。", + "question": "cause", + "label": "0" + } + ], + 'hi': [ + { + "premise": "M te vle konsève enèji.", + "choice1": "Mwen te fin baleye chanm lib la.", + "choice2": "Mwen te femen limyè nan chanm lib la.", + "question": "effect", + "label": "1" + }, + { + "premise": "Flam bouji a te etenn.", + "choice1": "Mwen te soufle bouji a.", + "choice2": "Mwen te limen mèch bouji a.", + "question": "cause", + "label": "0" + } + ] +} +``` +In this example, we format the examples use the non-verbal prompts `{premise}\n{choice1}` and `{premise}\n{choice2}`, which are shared by all three languages. +```python +from fairseq.models.transformer_lm import TransformerLanguageModel + +model_dir = 'path_to_decompressed_tar_gz_dir' +lm = TransformerLanguageModel.from_pretrained(model_dir, bpe='sentencepiece') +lm = lm.eval() +lm = lm.half() +lm = lm.cuda() + +def get_logprobs(prompt): + import re + prompt = re.sub('\n+' , '\n', prompt) # collapse repeated newlines, which indicate separate documents + return lm.score(prompt, replace_newlines_with_eos=True)['positional_scores'] + +# Zero-shot evaluation for the Choice of Plausible Alternatives (COPA) task. +# A return value of 0 indicates that the first alternative is more plausible, +# while 1 indicates that the second alternative is more plausible. +def COPA_eval(prompt, alternative1, alternative2): + lprob1 = get_logprobs(prompt + "\n" + alternative1).sum() + lprob2 = get_logprobs(prompt + "\n" + alternative2).sum() + return 0 if lprob1 > lprob2 else 1 + +for lang in ['en', 'zh', 'hi']: + for idx, example in enumerate(data_samples[lang]): + predict = COPA_eval(example["premise"], example["choice1"], example["choice2"]) + print(f'{lang}-{idx}', predict, example['label']) + +# en-0 1 1 +# en-1 0 0 +# zh-0 1 1 +# zh-1 0 0 +# hi-0 1 1 +# hi-1 0 0 +``` + +## XStoryCloze + +We release XStoryCloze, a new multilingual dataset intended for few-shot evaluation, alongside this paper. XStoryCloze consists of professional translation of the validation split of the [English StoryCloze dataset](https://cs.rochester.edu/nlp/rocstories/) (Spring 2016 version) to 10 other languages. It is opensourced under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/legalcode), the same license as the English StoryCloze. + +You can download the dataset via [this link](https://dl.fbaipublicfiles.com/xstorycloze.zip). + +Language | ar | es | eu | hi | id | my | ru | sw | te | zh +---|---|---|---|---|---|---|---|---|---|--- +Train size | 360 | 360 | 360 | 360 | 360 | 360 | 360 | 360 | 360 | 360 +Eval size | 1511 | 1511 | 1511 | 1511 | 1511 | 1511 | 1511 | 1511 | 1511 | 1511 + +Please refer to [the dataset doc](XStoryCloze.md) for more information. + + +## Publication +[Few-shot Learning with Multilingual Generative Language Models](https://arxiv.org/abs/2112.10668). +Xi Victoria Lin*, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li* (* Equal Contribution). +EMNLP 2022. + +## Citation +``` +@article{DBLP:journals/corr/abs-2112-10668, + author = {Xi Victoria Lin and + Todor Mihaylov and + Mikel Artetxe and + Tianlu Wang and + Shuohui Chen and + Daniel Simig and + Myle Ott and + Naman Goyal and + Shruti Bhosale and + Jingfei Du and + Ramakanth Pasunuru and + Sam Shleifer and + Punit Singh Koura and + Vishrav Chaudhary and + Brian O'Horo and + Jeff Wang and + Luke Zettlemoyer and + Zornitsa Kozareva and + Mona T. Diab and + Veselin Stoyanov and + Xian Li}, + title = {Few-shot Learning with Multilingual Language Models}, + journal = {CoRR}, + volume = {abs/2112.10668}, + year = {2021}, + url = {https://arxiv.org/abs/2112.10668}, + eprinttype = {arXiv}, + eprint = {2112.10668}, + timestamp = {Tue, 04 Jan 2022 15:59:27 +0100}, + biburl = {https://dblp.org/rec/journals/corr/abs-2112-10668.bib}, + bibsource = {dblp computer science bibliography, https://dblp.org} +} +``` diff --git a/data/fairseq/examples/xglm/XStoryCloze.md b/data/fairseq/examples/xglm/XStoryCloze.md new file mode 100644 index 0000000000000000000000000000000000000000..9b0fce07159ab0f4886acc206da4afd3b450f453 --- /dev/null +++ b/data/fairseq/examples/xglm/XStoryCloze.md @@ -0,0 +1,57 @@ +XStoryCloze consists of professional translation of the validation split of the [English StoryCloze dataset](https://cs.rochester.edu/nlp/rocstories/) (Spring 2016 version) to 10 other languages. This dataset is released by FAIR (Fundamental Artificial Intelligence Research) alongside the paper [Few-shot Learning with Multilingual Generative Language Models. EMNLP 2022](https://arxiv.org/abs/2112.10668). + +# Languages +ru, zh (Simplified), es (Latin America), ar, hi, id, te, sw, eu, my. + +# Data Splits +This dataset is intended to be used for evaluating the zero- and few-shot learning capabilities of multlingual language models. We split the data for each language into train and test (360 vs. 1510 examples, respectively). The released data files for different languages maintain a line-by-line alignment. + +# Access English StoryCloze +Please request the original English StoryCloze dataset through the [official website](https://cs.rochester.edu/nlp/rocstories/). You can create a split of the en data following our data split scheme using the following commands: +``` +head -361 spring2016.val.tsv > spring2016.val.en.tsv.split_20_80_train.tsv + +head -1 spring2016.val.tsv > spring2016.val.en.tsv.split_20_80_eval.tsv # TSV header +tail -1511 spring2016.val.tsv >> spring2016.val.en.tsv.split_20_80_eval.tsv +``` + +# Licence +XStoryCloze is opensourced under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/legalcode), the same license as the original English StoryCloze. + +# Citation +We hope this dataset is helpful for the research and wider NLP community. If you use XStoryCloze in your work, please cite +``` +@article{DBLP:journals/corr/abs-2112-10668, + author = {Xi Victoria Lin and + Todor Mihaylov and + Mikel Artetxe and + Tianlu Wang and + Shuohui Chen and + Daniel Simig and + Myle Ott and + Naman Goyal and + Shruti Bhosale and + Jingfei Du and + Ramakanth Pasunuru and + Sam Shleifer and + Punit Singh Koura and + Vishrav Chaudhary and + Brian O'Horo and + Jeff Wang and + Luke Zettlemoyer and + Zornitsa Kozareva and + Mona T. Diab and + Veselin Stoyanov and + Xian Li}, + title = {Few-shot Learning with Multilingual Language Models}, + journal = {CoRR}, + volume = {abs/2112.10668}, + year = {2021}, + url = {https://arxiv.org/abs/2112.10668}, + eprinttype = {arXiv}, + eprint = {2112.10668}, + timestamp = {Tue, 04 Jan 2022 15:59:27 +0100}, + biburl = {https://dblp.org/rec/journals/corr/abs-2112-10668.bib}, + bibsource = {dblp computer science bibliography, https://dblp.org} +} +``` diff --git a/data/fairseq/examples/xglm/model_card.md b/data/fairseq/examples/xglm/model_card.md new file mode 100644 index 0000000000000000000000000000000000000000..2656ec5d63ea288df744dbe2e1d77e52898183c2 --- /dev/null +++ b/data/fairseq/examples/xglm/model_card.md @@ -0,0 +1,152 @@ +# XGLM multilingual model +## Version 1.0.0 + +### Model developer +FAIR (Fundamental Artificial Intelligence Research) + +### Model type +A family of multilingual autoregressive language models (ranging from 564 million to 7.5 billion parameters) trained on a balanced corpus of a diverse set of languages. The language model can learn tasks from natural language descriptions and a few examples. + +### Model Feedback Channel +https://github.com/pytorch/fairseq + +## Intended use +### Primary intended use +For research purposes only, e.g. reproducing model evaluation results. Generation is only used in a limited capacity for explanation/justification or for prompting/probing/priming for class labels. + +### Out of scope uses +The primary purpose of the model is not to generate language, although the model is capable of doing that. + +## Potential risks +This section lists the potential risks associated with using the model. + +### Relevant factors +Based on known problems with NLP technology, potential relevant factors include output correctness, robustness, bias (gender, profession, race and religion), etc. + +### Evaluation factors +The model was evaluated on hate speech detection and occupation identification. +* Hate speech detection (Huang et al. (2020)) - A safety task to test language models’ ability to identify hateful and offensive text. +* Occupation identification (De-Arteaga et al., 2019), (Zhao et al., 2020) - A bias task to study language models’ performance divergence between different gender groups on the task of occupation identification. + +## Metrics +### Model performance measures +The XGLM model was primarily evaluated on +1. Zero shot and few shot learning by looking at per-language performance on tasks spanning commonsense reasoning (XCOPA, XWinograd), natural language inference (XNLI) and paraphrasing (PAWS-X). The model is also evaluated on XStoryCloze, a new dataset created by FAIR (Fundamental Artificial Intelligence Research). +2. Cross lingual transfer through templates and few-shot examples. +3. Knowledge probing - Evaluate to what extent the XGLM model can effectively store factual knowledge in different languages using the mLAMA benchmark. +4. Translation - We report machine translation results on WMT benchmarks and a subset of FLORES-101 in the main paper. + +The model was also evaluated on hate speech datasets introduced by Huang et al. (2020) and an occupation identification dataset by De-Arteaga et al. 2019 to identify bias in the model. + +### Approaches to handle uncertainty +Report confidence intervals, variance metrics for the model performance metrics. Few-shot evaluation was conducted with different sampling with 5 seeds. We reported statistical significance. + +## Evaluation data +## Zero Shot and Few Shot evaluation + +### XNLI (Conneau et al., 2018) +#### Description +The Cross-lingual Natural Language Inference (XNLI) corpus is the extension of the Multi-Genre NLI (MultiNLI) corpus to 15 languages. The dataset was created by manually translating the validation and test sets of MultiNLI into each of those 15 languages. + +### XStoryCloze +#### Description +A new dataset created by FAIR along side this work by translating the validation split of the English StoryCloze dataset (Mostafazadeh et al., 2016) (Spring 2016 version) to 10 other typologically diverse languages (ru, zh Simplified, es Latin America, ar, hi, id, te, sw, eu, my). + +### XCOPA (Ponti et al., 2020) +#### Description +The Cross-lingual Choice of Plausible Alternatives (XCOPA) dataset is a benchmark to evaluate the ability of machine learning models to transfer commonsense reasoning across languages. The dataset is the translation and reannotation of the English COPA (Roemmele et al. 2011) and covers 11 languages from 11 families and several areas around the globe. + +### XWinograd (Tikhonov and Ryabinin, 2021) +#### Description +XWinograd is a multilingual collection of Winograd Schemas in six languages that can be used for evaluation of cross-lingual commonsense reasoning capabilities. + +### PAWS-X (Yang et al., 2019) +#### Description +PAWS-X contains 23,659 human translated PAWS evaluation pairs and 296,406 machine translated training pairs in six typologically distinct languages: French, Spanish, German, Chinese, Japanese, and Korean. All translated pairs are sourced from examples in PAWS-Wiki. + +## Responsible AI (RAI) evaluation +### Hate speech (Huang et al. 2020) +This is a multilingual Twitter corpus for the task of hate speech detection with inferred four author demographic factors: age, country, gender and race/ethnicity. The corpus covers five languages: English, Italian, Polish, Portuguese and Spanish. + +### Bias dataset (De-Arteaga et al. 2019) +The aim of this dataset is to study the gender bias of models that identify a person’s occupation from their bios. + +---- + +## Training data +### CC100-XL +#### Description +Following the recent success of multilingual self-supervised pre-training (Devlin et al., 2019; Lample and Conneau, 2019; Con; Xue et al., 2020; Goyal et al., 2021a; Liu et al., 2020), we train our language models on a mixture of monolingual text of different languages. We extended the pipeline used for mining the CC100 corpus to generate CC100-XL, a significantly larger multilingual dataset covering 68 Common Crawl snapshots (from Summer 2013 to March/April 2020) and 134 languages. + +More details on the CC100-XL dataset can be found in the Appendix section of the paper. + +## RAI Dimensions +### Fairness (Bias and inclusion) +The XGLM model was evaluated on Hate speech and bias identification datasets. For hate speech, we observe that across the 5 languages in the dataset, in context learning results are only slightly better than random (50%). Another interesting observation is that most few shot results are worse than zero-shot, which indicates that the model is not able to utilize examples using the templates described in the paper. For bias identification, the XGLM (6.7B) English only model achieves the best performance on English and Spanish, while the GPT-3 model of comparable size (6.7B) model achieves the best in French. On certain occupations (e.g. model and teacher), XGLM 6.7B En only model and GPT-3 (6.7B) have very significant bias while XGLM 7.5B is much less biased. + +### Privacy and security +The XGLM model did not have any special Privacy and Security considerations. The training data and evaluation data were both public and went through standard Meta privacy and licensing procedures. + +### Transparency and control +In the spirit of transparency and accountability we have created this model card and a data card for the CC100-XL which can be found in the Appendix section of the paper. + +### Efficiency (Green AI) +From an engineering perspective, XGLM pertains to a family of models that represent single unified models catering to many languages which have wide application across many applications. Such a unified single model saves on carbon footprint as well as energy consumption (comparing to the alternative: separate models for different languages) leading to more energy efficiency. A single model, despite having the risk of being a single point of failure, has the powerful incentive of being easier to maintain, access, distribute, and track. + +## References +Edoardo Maria Ponti, Goran Glavas, Olga Majewska, Qianchu Liu, Ivan Vulic, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 2362–2376. Association for Computational Linguistics. +XCOPA Dataset | Papers With Code + +Alexey Tikhonov and Max Ryabinin. 2021. It’s all in the heads: Using attention heads as a baseline for cross-lingual transfer in commonsense reasoning. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 3534–3546. Association for Computational Linguistics. +XWINO Dataset | Papers With Code (XWinograd) + +Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. CoRR, abs/1908.11828. +PAWS-X Dataset | Papers With Code + +Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: evaluating cross-lingual sentence representations. CoRR, abs/1809.05053. +XNLI Dataset | Papers With Code + +Xiaolei Huang, Linzi Xing, Franck Dernoncourt, and Michael Paul. 2020. Multilingual twitter corpus and baselines for evaluating demographic bias in hate speech recognition. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1440–1448. + +Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In proceedings of the Conference on Fairness, Accountability, and Transparency, pages 120–128. + +Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, James F. Allen. A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories. CoRR abs/1604.01696. + +Jieyu Zhao, Subhabrata Mukherjee, Saghar Hosseini, Kai-Wei Chang, and Ahmed Hassan Awadallah. 2020. Gender bias in multilingual embeddings and crosslingual transfer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2896–2907. + +## Citation details +``` +@article{DBLP:journals/corr/abs-2112-10668, + author = {Xi Victoria Lin and + Todor Mihaylov and + Mikel Artetxe and + Tianlu Wang and + Shuohui Chen and + Daniel Simig and + Myle Ott and + Naman Goyal and + Shruti Bhosale and + Jingfei Du and + Ramakanth Pasunuru and + Sam Shleifer and + Punit Singh Koura and + Vishrav Chaudhary and + Brian O'Horo and + Jeff Wang and + Luke Zettlemoyer and + Zornitsa Kozareva and + Mona T. Diab and + Veselin Stoyanov and + Xian Li}, + title = {Few-shot Learning with Multilingual Language Models}, + journal = {CoRR}, + volume = {abs/2112.10668}, + year = {2021}, + url = {https://arxiv.org/abs/2112.10668}, + eprinttype = {arXiv}, + eprint = {2112.10668}, + timestamp = {Tue, 04 Jan 2022 15:59:27 +0100}, + biburl = {https://dblp.org/rec/journals/corr/abs-2112-10668.bib}, + bibsource = {dblp computer science bibliography, https://dblp.org} +} +```