Opusculum-1.5B / README.md
Stoned-Code's picture
Update README.md
b9e8db7 verified
|
Raw History Blame Contribute Delete
1.74 kB
metadata
datasets:
  - Stoned-Code/magnum-v4
  - Delta-Vector/Hydrus-Claude-Instruct-5K
  - anthracite-org/kalo-opus-instruct-22k-no-refusal
  - Epiculous/SynthRP-Gens-v1.1-Filtered-n-Cleaned
  - lodrick-the-lafted/kalo-opus-instruct-3k-filtered
  - anthracite-org/nopm_claude_writing_fixed
  - Epiculous/Synthstruct-Gens-v1.1-Filtered-n-Cleaned
  - anthracite-org/kalo_opus_misc_240827
  - anthracite-org/kalo_misc_part2
base_model:
  - unsloth/Qwen2.5-1.5B-bnb-4bit
pipeline_tag: text-generation
license: apache-2.0
language:
  - en
tags:
  - unsloth

A model finetuned from unsloth/Qwen2.5-1.5B-bnb-4bit using the datasets used to train anthracite-org/magnum-v4-9b.

The model is a full finetune using the following configuration on a single 5060ti with 16GB of VRAM.

training:
  max_seq_length: 8192
  num_epochs: 3
  learning_rate: 0.00002
  embedding_learning_rate: null
  batch_size: 2
  gradient_accumulation_steps: 4
  warmup_steps: 3
  max_steps: 0
  save_steps: 100
  eval_steps: 100
  early_stopping_patience: 0
  weight_decay: 0.001
  random_seed: 3407
  packing: true
  train_on_completions: true
  gradient_checkpointing: unsloth
  offload_layers: 0
  offload_vram_gb: null
  offload_vram_gb_per_device: {}
  prefetch_depth: 2
  optim: adamw_bnb_8bit
  lr_scheduler_type: cosine
lora:
  lora_r: 16
  lora_alpha: 16
  lora_dropout: 0
  target_modules:
    - q_proj
    - k_proj
    - v_proj
    - o_proj
    - gate_proj
    - up_proj
    - down_proj
  use_rslora: false
  use_loftq: false
  use_dora: false
logging:
  enable_wandb: false
  wandb_project: ''
  enable_tensorboard: false
  tensorboard_dir: ''
  log_frequency: 10