Fine-tuning-Gpt-2

LoRA adapter for openai-community/gpt2, fine-tuned with TRL SFT on TinyStories.

The base GPT-2 weights are unchanged. This repo contains the adapter only.

Training

• 500 steps on train[:8000], sequence length 256
• LoRA rank 16, alpha 32, dropout 0.05, targets c_attn, c_proj, c_fc
• Batch size 1, gradient accumulation 8, learning rate 2e-4
• Final train loss 2.145, eval loss 1.986

Trained locally on an Apple M3 Pro.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "openai-community/gpt2"
adapter_id = "omaralshareef/Fine-tuning-Gpt-2"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id)
model = PeftModel.from_pretrained(model, adapter_id)

prompt = "Once upon a time there was a little girl named Lucy."
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=80, do_sample=True, temperature=0.8, top_p=0.95)
print(tokenizer.decode(output[0], skip_special_tokens=True))