July 8:
- link to HDFS 
- HDFS address: /mnt/hdfs/pllm/yutong
- uploaded two benchmarks 
- downloaded the long_lamp datasets

HDFS删除文件： hadoop fs -rm -r /hdfs/文件夹/路径
删除单个文件： hadoop fs -rm /hdfs/文件/路径

baseline training for abtract genration:


PYTHONPATH=. python3 train_llm.py \
  --train_data /mnt/hdfs/pllm/yutong/long_lamp/abstract_train.json \
  --validation_data /mnt/hdfs/pllm/yutong/long_lamp/abstract_train.json \
  --test_data /mnt/hdfs/pllm/yutong/long_lamp/abstract_test.json \
  --model_name google/flan-t5-base \
  --task generation_abstract \
  --output_dir /mnt/hdfs/pllm/yutong/long_output

#多卡训练
torchrun --nproc_per_node=4 --master_port=29502 train_llm.py \
  --train_data /mnt/hdfs/pllm/yutong/long_lamp/abstract_train.json \
  --validation_data /mnt/hdfs/pllm/yutong/long_lamp/abstract_train.json \
  --test_data /mnt/hdfs/pllm/yutong/long_lamp/abstract_test.json \
  --model_name google/flan-t5-base \
  --task generation_abstract \
  --output_dir /mnt/hdfs/pllm/yutong/long_output


#改成llama 3.1 8B 版本
PYTHONPATH=. python3 train_llama.py \
  --train_data /mnt/hdfs/pllm/yutong/long_lamp/abstract_train.json \
  --validation_data /mnt/hdfs/pllm/yutong/long_lamp/abstract_train.json \
  --test_data /mnt/hdfs/pllm/yutong/long_lamp/abstract_test.json \
  --model_name meta-llama/Meta-Llama-3.1-8B\
  --task generation_abstract \
  --output_dir /mnt/hdfs/pllm/yutong/long_output_llama


pip install huggingface_hub
huggingface-cli login
huggingface token: <HF_TOKEN_REDACTED>
https://llama3-1.llamameta.net/*?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiNWUwOGc5MGRmY3hxcWN3eG5pOWtod2hxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvbGxhbWEzLTEubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTc1MjE5NDMxMX19fV19&Signature=JYdafzZYXEVmrFyp3DASI%7EdmID5eLnR%7EW0UonzxCEEdKymy6U2UYvHfA5xLIWY1oupVhRtLhVmy6Ro1UdfFYDLKGVGUed3piEbFMSfAj4jsycBXx9dHoAG4cwtq6kjMl50lNnD4PCGBV3IKO3hHyCSJ8GTYNaROGo9u2qOr6%7E--iCAI8jGv4jKLHjcR4flR9omhkMLxD7-g9L96PN8wEEkupayDmqCXk4lZ1KOX-h%7EityoB4m1-J5Y-zZVNUBti0My3hxrPTNn79Zs8v8-bORN1384OY3LgwMVK3CBTewUODGa6L4HKvRdY6mgATk9x06Gw-CdWLcoiJczpG44r4EA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=24251018147862817

HDFS 删除文件： hdfs dfs -rm -r harunava/home/byte_tiktok_content_ecology/user/yutong
/mnt/hdfs/pllm/abstract_train.json
hdfs dfs -rm /mnt/hdfs/pllm/abstract_test.json


HDFS 删除文件： hdfs dfs -rm -r harunava/home/byte_tiktok_content_ecology/user/yutong
/mnt/hdfs/pllm/abstract_train.json
hdfs dfs -rm /mnt/hdfs/pllm/abstract_test.json


July 16:
- baseline model using Llama 3.1
- model 2: RL for retrieval refinement 


July 17:
- 重新下载lamp数据集
- run long- writer在long_lamp数据集上