---
license: mit
datasets:
- roneneldan/TinyStories
language:
- en
base_model: gpt2
pipeline_tag: text-generation
---

# TinyTale-15M

TinyTale-15M is a 15 Million parameter custom GPT-2 architecture trained completely from scratch on the `TinyStories` dataset. This model serves as a baseline pipeline experiment for handling custom architecture initialization, offline dataset tokenization, and processing on hardware acceleration via an NVIDIA A10 GPU workstation.

## Model Architecture Specifications
- **Architecture:** Custom GPT-2 (with Tied Weights)
- **Layers (Blocks):** 6
- **Attention Heads:** 6
- **Embedding Dimension:** 384
- **Context Window:** 256 tokens
- **Unique Trainable Parameters:** ~15 Million (30.04M raw un-tied)

## Training Hyperparameters
- **Hardware:** 1x NVIDIA A10 GPU (24GB VRAM)
- **Precision:** FP16 mixed precision
- **Batch Size:** 64
- **Learning Rate:** 5e-4 (with linear decay)
- **Training Steps:** 3,000 steps
- **Dataset Subset:** 200,000 unique stories from `roneneldan/TinyStories`

## How to Use

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("agentbyumer/TinyTale-15M")
model = AutoModelForCausalLM.from_pretrained("agentbyumer/TinyTale-15M")

prompt = "Once upon a time, a small puppy found a shiny key"
inputs = tokenizer(prompt, return_tensors="pt")

output = model.generate(**inputs, max_length=100, do_sample=True, temperature=0.8)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```