---
license: mit
base_model: gpt2
tags:
  - text-generation
  - causal-lm
  - gpt
  - transformer
  - decoder-only
  - tiny-stories
  - story
  - children
  - stories
  - narrative
  - lm
  - language-model
  - pytorch
  - 29m
  - warmth
  - wisdom
  - inept-vocab
  - story-generation
  - causal-lm-pretraining
language:
  - en
pipeline_tag: text-generation
widget:
  - text: "One day, a little girl named Lily"
---

# StoryGPT

A small [GPT](https://arxiv.org/abs/1710.11815)-style causal language model trained from scratch on a 50k excerpt of the [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) dataset.

## Model Details

- **Architecture:** decoder-only transformer with causal self-attention and pre-LN blocks
- **Parameters:** ~29M
- **Vocabulary:** GPT-2 tokenizer (50,257 tokens)
- **Context length:** 512
- **Layers:** 4, **Embedding dim:** 256

## Training

- **Dataset:** TinyStories, 50,000 stories (44 MB of text)
- **Block size:** 128 tokens
- **Batch size:** 16
- **Optimizer:** AdamW, lr = 3e-4
- **Loss:** Cross-entropy (causal LM)

## Usage

Direct loading with `transformers` — no local files needed:

```python
from transformers import GPT2Tokenizer, AutoConfig, AutoModel

tokenizer = GPT2Tokenizer.from_pretrained("coderian/StoryGPT")

model = AutoModel.from_pretrained("coderian/StoryGPT")
model.eval()

text = "One day, a little girl named Lily"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(input_ids=inputs["input_ids"], max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

Because `StoryGPT` is a custom model, register its config class once if autoloading fails:

```python
from transformers import AutoConfig, AutoModel
from config_gpt import GPTConfig
from models import StoryGPT

AutoConfig.register("my_gpt", GPTConfig)
AutoModel.register(GPTConfig, StoryGPT)

model = AutoModel.from_pretrained("coderian/StoryGPT")
```

## Limitations

Tiny model trained for only 2 epochs. Stories may be short, repetitive, or contain small errors in grammar or logic.