GPT-2 Pirate πŸ΄β€β˜ οΈ

GPT-2 small (124M) post-trained to answer questions like a pirate.

Try it: GPT-2 Pirate demo, which shows plain GPT-2 and this model side by side.

Before vs after (questions not in the training data)

Question Plain GPT-2 GPT-2 Pirate
What is the capital of Japan? "The capital of Japan is Japan. Source: http://www.c-n.co.jp/..." "Arr, the capital be Tokyo, matey, the capital of Japan be the capital o' the Pacific!"
Why do birds sing? Rambles about birds' tongues "Arr, birds sing by day, matey, to keep the sun at bay, an' keep the wind away!"

Training

  • Data: 60 hand-written question / pirate-answer pairs
  • Method: full fine-tune (all 124M weights), AdamW, learning rate 5e-5, batch size 8, 6 epochs
  • Loss: computed on the answer tokens only (question tokens masked with -100)
  • Loss curve: 3.45 β†’ 1.10
  • Weight change: under 0.5% in any layer, yet the model picks up a new style and format

Usage

The model expects this prompt format:

from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("kruxarth/gpt2-pirate")
model = AutoModelForCausalLM.from_pretrained("kruxarth/gpt2-pirate")

prompt = "Question: How do I make tea?\nAnswer:"
inputs = tok(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=60, do_sample=True, temperature=0.7, top_p=0.9,
                     pad_token_id=tok.eos_token_id)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Limitations

This is a learning project. 60 examples teach the model a style, not knowledge, so answers are often factually wrong. GPT-2 small knows very little to begin with. Post-training shapes how a model answers; the knowledge comes from pretraining.

Downloads last month
45
Safetensors
Model size
0.1B params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for kruxarth/gpt2-pirate

Quantized
(101)
this model

Space using kruxarth/gpt2-pirate 1