| # AnyMo Project |
|
|
| # Dependencies and Installation |
| ```bash |
| pip install -r requirements.txt |
| pip install sentence_transformers==5.3.0 |
| |
| # 多机è®ç»ƒ |
| apt install perftest infiniband-diags libibverbs-dev ibutils rdma-core ibverbs-utils -y |
| pip install deepspeed |
| ``` |
|
|
| Download Pre-trained Models and put these models under `checkpoints` folder. |
|
|
| ```text |
| checkpoints/ |
| ├── distilbert-base-uncased |
| ├── flan-t5-xl |
| └── sentence-transformers-paraphrase-MiniLM-L6-v2 |
| ``` |
|
|
| # Prepare Data |
|
|
|
|
| tar -cf - work_dirs/ \ |
| | pv \ |
| | pigz -p 64 \ |
| | split -b 16G - work_dirs.tar.gz.part_ |
|
|
| tar -cf - work_dirs | pv | pigz -p 24 > work_dirs.tar.gz |
|
|
|
|
