AnyMo Project

Dependencies and Installation

pip install -r requirements.txt
pip install sentence_transformers==5.3.0

# 多机训练
apt install perftest infiniband-diags libibverbs-dev ibutils rdma-core ibverbs-utils -y
pip install deepspeed

Download Pre-trained Models and put these models under checkpoints folder.

checkpoints/
├── distilbert-base-uncased
├── flan-t5-xl
└── sentence-transformers-paraphrase-MiniLM-L6-v2

Prepare Data

tar -cf - work_dirs/
| pv
| pigz -p 64
| split -b 16G - work_dirs.tar.gz.part_

tar -cf - work_dirs | pv | pigz -p 24 > work_dirs.tar.gz