File size: 1,562 Bytes
a570a9c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 | ---
license: mit
datasets:
- roneneldan/TinyStories
language:
- en
base_model: gpt2
pipeline_tag: text-generation
---
# TinyTale-15M
TinyTale-15M is a 15 Million parameter custom GPT-2 architecture trained completely from scratch on the `TinyStories` dataset. This model serves as a baseline pipeline experiment for handling custom architecture initialization, offline dataset tokenization, and processing on hardware acceleration via an NVIDIA A10 GPU workstation.
## Model Architecture Specifications
- **Architecture:** Custom GPT-2 (with Tied Weights)
- **Layers (Blocks):** 6
- **Attention Heads:** 6
- **Embedding Dimension:** 384
- **Context Window:** 256 tokens
- **Unique Trainable Parameters:** ~15 Million (30.04M raw un-tied)
## Training Hyperparameters
- **Hardware:** 1x NVIDIA A10 GPU (24GB VRAM)
- **Precision:** FP16 mixed precision
- **Batch Size:** 64
- **Learning Rate:** 5e-4 (with linear decay)
- **Training Steps:** 3,000 steps
- **Dataset Subset:** 200,000 unique stories from `roneneldan/TinyStories`
## How to Use
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("agentbyumer/TinyTale-15M")
model = AutoModelForCausalLM.from_pretrained("agentbyumer/TinyTale-15M")
prompt = "Once upon a time, a small puppy found a shiny key"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_length=100, do_sample=True, temperature=0.8)
print(tokenizer.decode(output[0], skip_special_tokens=True))
``` |