Opusculum-1.5B / README.md
Stoned-Code's picture
Update README.md
b9e8db7 verified
|
Raw History Blame Contribute Delete
1.74 kB
---
datasets:
- Stoned-Code/magnum-v4
- Delta-Vector/Hydrus-Claude-Instruct-5K
- anthracite-org/kalo-opus-instruct-22k-no-refusal
- Epiculous/SynthRP-Gens-v1.1-Filtered-n-Cleaned
- lodrick-the-lafted/kalo-opus-instruct-3k-filtered
- anthracite-org/nopm_claude_writing_fixed
- Epiculous/Synthstruct-Gens-v1.1-Filtered-n-Cleaned
- anthracite-org/kalo_opus_misc_240827
- anthracite-org/kalo_misc_part2
base_model:
- unsloth/Qwen2.5-1.5B-bnb-4bit
pipeline_tag: text-generation
license: apache-2.0
language:
- en
tags:
- unsloth
---
A model finetuned from [unsloth/Qwen2.5-1.5B-bnb-4bit](https://huggingface.co/unsloth/Qwen2.5-1.5B-bnb-4bit) using the datasets used to train [anthracite-org/magnum-v4-9b](https://huggingface.co/anthracite-org/magnum-v4-9b).
The model is a full finetune using the following configuration on a single 5060ti with 16GB of VRAM.
```yaml
training:
max_seq_length: 8192
num_epochs: 3
learning_rate: 0.00002
embedding_learning_rate: null
batch_size: 2
gradient_accumulation_steps: 4
warmup_steps: 3
max_steps: 0
save_steps: 100
eval_steps: 100
early_stopping_patience: 0
weight_decay: 0.001
random_seed: 3407
packing: true
train_on_completions: true
gradient_checkpointing: unsloth
offload_layers: 0
offload_vram_gb: null
offload_vram_gb_per_device: {}
prefetch_depth: 2
optim: adamw_bnb_8bit
lr_scheduler_type: cosine
lora:
lora_r: 16
lora_alpha: 16
lora_dropout: 0
target_modules:
- q_proj
- k_proj
- v_proj
- o_proj
- gate_proj
- up_proj
- down_proj
use_rslora: false
use_loftq: false
use_dora: false
logging:
enable_wandb: false
wandb_project: ''
enable_tensorboard: false
tensorboard_dir: ''
log_frequency: 10
```