There is a change in bert and gpt2 codebase
Eval file for tinygsm:
/work/jf381/code/lm-research/scripts/training/train_medium_bash_resume.sh

1, For GPT2:
We have gpt2 tokenizer: need tcohange
we will have some files to modify in /work/jf381/code/lm-research/scripts/training/train_medium_bash_resume.sh

FST_353M has some files trained with old version resume_new
FST_1_3B is up to date resume
Transformer_1_3B is up to date resume
Transformer_353M is up to date resume

2, 

/work/jf381/code/lm-research/scripts/training/train_medium_bash_resume_transformer_bert_prediction.sh

For Bert:
we will start from 2 gpu version of ar model trained on slim-6B
We have bert tokenizer: need to change 
bert_2_gpu_transformer
bert_2_gpu_fst