anthracite-org/nopm_claude_writing_fixed
Viewer • Updated • 6.35k • 1.3k • 20
A model finetuned from unsloth/Qwen2.5-1.5B-bnb-4bit using the datasets used to train anthracite-org/magnum-v4-9b.
The model is a full finetune using the following configuration on a single 5060ti with 16GB of VRAM.
training:
max_seq_length: 8192
num_epochs: 3
learning_rate: 0.00002
embedding_learning_rate: null
batch_size: 2
gradient_accumulation_steps: 4
warmup_steps: 3
max_steps: 0
save_steps: 100
eval_steps: 100
early_stopping_patience: 0
weight_decay: 0.001
random_seed: 3407
packing: true
train_on_completions: true
gradient_checkpointing: unsloth
offload_layers: 0
offload_vram_gb: null
offload_vram_gb_per_device: {}
prefetch_depth: 2
optim: adamw_bnb_8bit
lr_scheduler_type: cosine
lora:
lora_r: 16
lora_alpha: 16
lora_dropout: 0
target_modules:
- q_proj
- k_proj
- v_proj
- o_proj
- gate_proj
- up_proj
- down_proj
use_rslora: false
use_loftq: false
use_dora: false
logging:
enable_wandb: false
wandb_project: ''
enable_tensorboard: false
tensorboard_dir: ''
log_frequency: 10