File size: 638 Bytes
8b3f571 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | # AnyMo Project
# Dependencies and Installation
```bash
pip install -r requirements.txt
pip install sentence_transformers==5.3.0
# 多机è®ç»ƒ
apt install perftest infiniband-diags libibverbs-dev ibutils rdma-core ibverbs-utils -y
pip install deepspeed
```
Download Pre-trained Models and put these models under `checkpoints` folder.
```text
checkpoints/
├── distilbert-base-uncased
├── flan-t5-xl
└── sentence-transformers-paraphrase-MiniLM-L6-v2
```
# Prepare Data
tar -cf - work_dirs/ \
| pv \
| pigz -p 64 \
| split -b 16G - work_dirs.tar.gz.part_
tar -cf - work_dirs | pv | pigz -p 24 > work_dirs.tar.gz
|