--- datasets: - Stoned-Code/magnum-v4 - Delta-Vector/Hydrus-Claude-Instruct-5K - anthracite-org/kalo-opus-instruct-22k-no-refusal - Epiculous/SynthRP-Gens-v1.1-Filtered-n-Cleaned - lodrick-the-lafted/kalo-opus-instruct-3k-filtered - anthracite-org/nopm_claude_writing_fixed - Epiculous/Synthstruct-Gens-v1.1-Filtered-n-Cleaned - anthracite-org/kalo_opus_misc_240827 - anthracite-org/kalo_misc_part2 base_model: - unsloth/Qwen2.5-1.5B-bnb-4bit pipeline_tag: text-generation license: apache-2.0 language: - en tags: - unsloth --- A model finetuned from [unsloth/Qwen2.5-1.5B-bnb-4bit](https://huggingface.co/unsloth/Qwen2.5-1.5B-bnb-4bit) using the datasets used to train [anthracite-org/magnum-v4-9b](https://huggingface.co/anthracite-org/magnum-v4-9b). The model is a full finetune using the following configuration on a single 5060ti with 16GB of VRAM. ```yaml training: max_seq_length: 8192 num_epochs: 3 learning_rate: 0.00002 embedding_learning_rate: null batch_size: 2 gradient_accumulation_steps: 4 warmup_steps: 3 max_steps: 0 save_steps: 100 eval_steps: 100 early_stopping_patience: 0 weight_decay: 0.001 random_seed: 3407 packing: true train_on_completions: true gradient_checkpointing: unsloth offload_layers: 0 offload_vram_gb: null offload_vram_gb_per_device: {} prefetch_depth: 2 optim: adamw_bnb_8bit lr_scheduler_type: cosine lora: lora_r: 16 lora_alpha: 16 lora_dropout: 0 target_modules: - q_proj - k_proj - v_proj - o_proj - gate_proj - up_proj - down_proj use_rslora: false use_loftq: false use_dora: false logging: enable_wandb: false wandb_project: '' enable_tensorboard: false tensorboard_dir: '' log_frequency: 10 ```