---
language:
- en
license: mit
tags:
- gpt2
- tiny-llm
- educational
- pytorch
datasets:
- HuggingFaceFW/fineweb-edu
metrics:
- perplexity
pipeline_tag: text-generation
---

# Tiny-LLM-From-Scratch (~50M)

A ~50M parameter GPT-2-style decoder-only Transformer built **entirely from scratch** in PyTorch
and pre-trained on [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu).

This model was created for educational purposes to demonstrate how a GPT-style LLM works
at the tensor/code level.  Every component—tokenizer, data pipeline, attention, MLP,
training loop, and generation—is implemented from scratch without using the `transformers` library.

## Architecture Details

| Parameter | Value |
|---|---|
| **Architecture** | Decoder-only Transformer (Pre-LayerNorm, GPT-2 style) |
| **Total parameters** | 50,677,760 (50.68 M) |
| **Context length** | 256 tokens |
| **Vocabulary** | 16,384 Byte-Level BPE tokens |
| **Embedding dimension (C)** | 640 |
| **Attention heads (H)** | 10 (head_dim = 64) |
| **Transformer layers (N)** | 6 |
| **FFN hidden dim** | 2,560 (4 x C) |
| **Activation** | GELU |
| **Weight tying** | No |

## Training Details

| Setting | Value |
|---|---|
| **Dataset** | FineWeb-Edu sample-10BT |
| **Training tokens** | ~20 M |
| **Tokenizer** | Custom Byte-Level BPE (trained on 80k docs) |
| **Optimiser** | AdamW (lr=3e-4, weight_decay=0.1, betas=(0.9, 0.95)) |
| **LR schedule** | Cosine with 500-step linear warm-up |
| **Effective batch size** | 128 sequences (32 × 4 grad accum) |
| **Steps** | 5,000 |
| **AMP** | FP16 (PyTorch autocast + GradScaler) |
| **Hardware** | Google Colab T4 (16 GB) |

## Sampling Capabilities

Supports autoregressive text generation with:
- **Temperature** scaling
- **Top-K** filtering
- **Nucleus (Top-P)** sampling

## Usage (PyTorch)

```python
from huggingface_hub import hf_hub_download
import torch, json

# Download files
config_path = hf_hub_download(repo_id="tiny-llm-50m", filename="config.json")
weights_path = hf_hub_download(repo_id="tiny-llm-50m", filename="pytorch_model.bin")

# Rebuild model (copy GPTConfig and GPT classes from the training notebook)
with open(config_path) as f:
    cfg_dict = json.load(f)
config = GPTConfig(**cfg_dict)
model  = GPT(config)
model.load_state_dict(torch.load(weights_path, map_location="cpu"))
model.eval()
```

## Source Code

Full project source: [Tiny-LLM-From-Scratch](https://github.com/madhurchouhan7/tiny-llm)