-
Pushed weights for checkpoint 150 to test on SEB
Checkpoint 100 achieved mean score of 56.2% on SEB and outperformed all previous models. step:150 might be overfitting. -
Copied model tensors from checkpoint 100 into main root
-
Supervised finetuning of LLM2Vec-da best performing SiMCSE model
Trained on this dataset https://huggingface.co/datasets/DDSC/da-wikipedia-queries-gemma-processed / synthetic queries generated using Gemma on DK wiki articles
Training config:
{
"model_name_or_path": "AI-Sweden-Models/Llama-3-8B-instruct",
"peft_model_name_or_path": "jealk/llm2vec-da-simcse",
"bidirectional": true,
"pooling_mode": "mean",
"dataset_name": "DKData",
"dataset_file_path": "cache/dk-data",
"remove_unused_columns": false,
"learning_rate": 2e-4,
"num_train_epochs": 3,
"warmup_steps": 300,
"per_device_train_batch_size": 128,
"per_device_eval_batch_size": 128,
"gradient_accumulation_steps": 4,
"do_train": true,
"disable_tqdm": false,
"max_seq_length": 256,
"overwrite_output_dir": true,
"output_dir": "output/mntp-supervised/Meta-Llama-3-8B-Swe-DK-data-2",
"logging_steps": 5,
"save_steps": 10,
"save_only_model": true,
"stop_after_n_steps": 1000,
"lora_r": 16,
"gradient_checkpointing": true,
"torch_dtype": "bfloat16",
"attn_implementation": "flash_attention_2",
"seed": 42,
"exp… -
initial commit
