Built with Axolotl

See axolotl config

axolotl version: 0.17.0

# Eugeny v0.1 seed SFT โ€” Axolotl canonical config
# Binding stack: W&B + SkyPilot + Axolotl only.

base_model: Qwen/Qwen3.6-27B
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
trust_remote_code: true

# Full bf16 LoRA, not 4-bit QLoRA.
load_in_8bit: false
load_in_4bit: false
strict: false
bf16: true
fp16: false
tf32: true

# Dataset: OpenAI-style messages JSONL.
chat_template: tokenizer_default
    type: chat_template
    field_messages: messages
    roles_to_train:
      - assistant
    train_on_eos: turn

test_datasets:
  - path: ./fine-tuning/data/seed/sft_eval.jsonl
    type: chat_template
    field_messages: messages
    roles_to_train:
      - assistant
    train_on_eos: turn
    split: train

val_set_size: 0.0
dataset_prepared_path: ./fine-tuning/data/seed/axolotl_prepared_qwen36_27b_seed

# Sequence / packing.
sequence_len: 4096
sample_packing: false
pad_to_sequence_len: true

# LoRA.
adapter: lora
lora_model_dir:
lora_r: 64
lora_alpha: 128
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:

# Optimization.
output_dir: ./outputs/qwen36-27b-agent-memory-sft-lora-axolotl
num_epochs: 2
micro_batch_size: 1
gradient_accumulation_steps: 16
learning_rate: 0.0001
optimizer: adamw_torch
lr_scheduler: cosine
warmup_ratio: 0.03
weight_decay: 0.01
gradient_checkpointing: true

# Logging/checkpoints.
logging_steps: 5
eval_steps: 200
save_steps: 200
save_total_limit: 2
saves_per_epoch:
seed: 42

# W&B is mandatory for production runs.
wandb_project: eugeny
wandb_name: eugeny-qwen36-27b-seed-sft
wandb_log_model: checkpoint

# Hub upload. HF token is provided by SkyPilot secret/env.
hub_model_id: fornace-ml/eugeny-seed-adapters
hub_strategy: every_save
push_to_hub: true

# Metadata for humans and downstream gates.
metadata:
  product: eugeny
  run_type: seed_sft
  intended_behavior: "Bake constitutional voice + allow/refuse boundary + memory discipline across 5 languages."
  dataset_manifest: ./fine-tuning/data/DATASET_MANIFEST.md
  base_model: Qwen/Qwen3.6-27B
  tracking: wandb
  orchestration: skypilot
  training_framework: axolotl

Visualize in Weights & Biases

eugeny-seed-adapters

This model is a fine-tuned version of Qwen/Qwen3.6-27B on the ./fine-tuning/data/seed/sft_train.jsonl dataset. It achieves the following results on the evaluation set:

  • Loss: 0.8184
  • Ppl: 2.2668
  • Memory/max Active (gib): 66.74
  • Memory/max Allocated (gib): 66.74
  • Memory/device Reserved (gib): 75.09

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0001
  • train_batch_size: 1
  • eval_batch_size: 1
  • seed: 42
  • gradient_accumulation_steps: 16
  • total_train_batch_size: 16
  • optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_steps: 2
  • training_steps: 45

Training results

Training Loss Epoch Step Validation Loss Ppl Active (gib) Allocated (gib) Reserved (gib)
No log 0 0 2.2034 9.0557 63.23 63.23 63.62
0.5623 1.9805 45 0.8184 2.2668 66.74 66.74 75.09

Framework versions

  • PEFT 0.19.1
  • Transformers 5.9.0
  • Pytorch 2.12.1+cu130
  • Datasets 4.8.5
  • Tokenizers 0.22.2
Downloads last month
4
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for fornace-ml/eugeny-seed-adapters

Base model

Qwen/Qwen3.6-27B
Adapter
(592)
this model