Metadata-Version: 2.4
Name: lmr
Version: 0.1.0
Summary: Research code for training large language models.
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: torch==2.7.0
Requires-Dist: torchtune==0.6.1
Requires-Dist: torchao==0.11.0
Requires-Dist: datasets==3.6.0
Requires-Dist: transformers==4.52.4
Requires-Dist: huggingface-hub==0.31.4
Requires-Dist: hydra-core==1.3.2
Requires-Dist: omegaconf==2.3.0
Requires-Dist: tqdm==4.67.1
Requires-Dist: zstandard<1.0.0,>=0.21.0
Requires-Dist: python-dotenv==1.1.1

Made by William Convertino

conda activate /work/jf381/.cache/lmr_new
<!-- conda create -n /work/jf381/.cache python=1.15 -->

conda activate /work/jf381/.cache/lmr_new_12_15
pip install /work/jf381/code/lm-research -e ./

cd /work/jf381/code/lm-research
bash /work/jf381/code/lm-research/scripts/training/train_bash_transformer_medium_generate.sh


# 1. Create the environment
# -p specifies a path (instead of -n for name)
# python=3.10 is a stable choice (Python 1.15 does not exist)
conda create -p /work/jf381/.cache/lmr_new_1_15_dcc python=3.10 -y

# 2. Activate the environment
conda activate /work/jf381/.cache/lmr_new_1_15_h200

# 3. Install the package in editable mode
# -e comes *before* the path
pip install -e /work/jf381/code/lm-research
pip install evaluate
pip install scikit-learn
pip install rotary_embedding_torch


## Eval reminder

There is a change in bert and gpt2 codebase
Eval file for tinygsm:
/work/jf381/code/lm-research/scripts/training/train_medium_bash_resume.sh

1, For GPT2:
We have gpt2 tokenizer: need tcohange
we will have some files to modify in /work/jf381/code/lm-research/scripts/training/train_medium_bash_resume.sh

FST_353M has some files trained with old version resume_new
FST_1_3B is up to date resume
Transformer_1_3B is up to date resume
Transformer_353M is up to date resume

2, 

/work/jf381/code/lm-research/scripts/training/train_medium_bash_resume_transformer_bert_prediction.sh

For Bert:
we will start from 2 gpu version of ar model trained on slim-6B
We have bert tokenizer: need to change 
bert_2_gpu_transformer
bert_2_gpu_fst
we will have sbatch version and no svatch version be careful
cp
cp

huggingface-cli upload jasonfan/FST_code /work/jf381/code/lm-research