Update README.md
Browse files
README.md
CHANGED
|
@@ -9,43 +9,7 @@ The training process for this cybersecurity application utilizes Meta-Llama-3-8B
|
|
| 9 |
- **Model type:** Llama3
|
| 10 |
- **Language(s) (NLP):** English
|
| 11 |
- **Configuration:**
|
| 12 |
-
model_name_or_path: meta-llama/Meta-Llama-3-8B-Instruct
|
| 13 |
-
quantization_bit: 4
|
| 14 |
-
quantization_method: bitsandbytes
|
| 15 |
|
| 16 |
-
### method
|
| 17 |
-
stage: sft\\
|
| 18 |
-
do_train: true\\
|
| 19 |
-
finetuning_type: lora\\
|
| 20 |
-
lora_target: all\\
|
| 21 |
-
|
| 22 |
-
### dataset
|
| 23 |
-
dataset: alpaca_en_demo,cybersecurity\\
|
| 24 |
-
template: llama3\\
|
| 25 |
-
cutoff_len: 2048\\
|
| 26 |
-
max_samples: 1000\\
|
| 27 |
-
overwrite_cache: true\\
|
| 28 |
-
preprocessing_num_workers: 16\\
|
| 29 |
-
|
| 30 |
-
### output
|
| 31 |
-
output_dir: saves/llama3-8b/lora/sft\\
|
| 32 |
-
logging_steps: 10\\
|
| 33 |
-
save_steps: 500\\
|
| 34 |
-
plot_loss: true\\
|
| 35 |
-
overwrite_output_dir: true\\
|
| 36 |
-
|
| 37 |
-
### train
|
| 38 |
-
per_device_train_batch_size: 1\\
|
| 39 |
-
gradient_accumulation_steps: 8\\
|
| 40 |
-
learning_rate: 1.0e-4\\
|
| 41 |
-
num_train_epochs: 3.0\\
|
| 42 |
-
lr_scheduler_type: cosine\\
|
| 43 |
-
warmup_ratio: 0.1\\
|
| 44 |
-
bf16: true\\
|
| 45 |
-
ddp_timeout: 180000000\\
|
| 46 |
-
|
| 47 |
-
### eval
|
| 48 |
-
val_size: 0.1\\
|
| 49 |
-
per_device_eval_batch_size: 1\\
|
| 50 |
-
eval_strategy: steps\\
|
| 51 |
-
eval_steps: 500\\
|
|
|
|
| 9 |
- **Model type:** Llama3
|
| 10 |
- **Language(s) (NLP):** English
|
| 11 |
- **Configuration:**
|
| 12 |
+
-model_name_or_path: meta-llama/Meta-Llama-3-8B-Instruct
|
| 13 |
+
-quantization_bit: 4
|
| 14 |
+
-quantization_method: bitsandbytes
|
| 15 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|