| ---
|
| license: mit
|
| datasets:
|
| - roneneldan/TinyStories
|
| language:
|
| - en
|
| base_model: gpt2
|
| pipeline_tag: text-generation
|
| ---
|
|
|
| # TinyTale-15M
|
|
|
| TinyTale-15M is a 15 Million parameter custom GPT-2 architecture trained completely from scratch on the `TinyStories` dataset. This model serves as a baseline pipeline experiment for handling custom architecture initialization, offline dataset tokenization, and processing on hardware acceleration via an NVIDIA A10 GPU workstation.
|
|
|
| ## Model Architecture Specifications
|
| - **Architecture:** Custom GPT-2 (with Tied Weights)
|
| - **Layers (Blocks):** 6
|
| - **Attention Heads:** 6
|
| - **Embedding Dimension:** 384
|
| - **Context Window:** 256 tokens
|
| - **Unique Trainable Parameters:** ~15 Million (30.04M raw un-tied)
|
|
|
| ## Training Hyperparameters
|
| - **Hardware:** 1x NVIDIA A10 GPU (24GB VRAM)
|
| - **Precision:** FP16 mixed precision
|
| - **Batch Size:** 64
|
| - **Learning Rate:** 5e-4 (with linear decay)
|
| - **Training Steps:** 3,000 steps
|
| - **Dataset Subset:** 200,000 unique stories from `roneneldan/TinyStories`
|
|
|
| ## How to Use
|
|
|
| ```python
|
| from transformers import AutoTokenizer, AutoModelForCausalLM
|
|
|
| tokenizer = AutoTokenizer.from_pretrained("agentbyumer/TinyTale-15M")
|
| model = AutoModelForCausalLM.from_pretrained("agentbyumer/TinyTale-15M")
|
|
|
| prompt = "Once upon a time, a small puppy found a shiny key"
|
| inputs = tokenizer(prompt, return_tensors="pt")
|
|
|
| output = model.generate(**inputs, max_length=100, do_sample=True, temperature=0.8)
|
| print(tokenizer.decode(output[0], skip_special_tokens=True))
|
| ``` |
