| --- |
| language: en |
| license: mit |
| tags: |
| - micro-gpt |
| - tinystories |
| - character-level |
| - pytorch |
| - transformer |
| - from-scratch |
| datasets: |
| - roneneldan/TinyStories |
| library_name: pytorch |
| --- |
| |
| # MicroGPT - Tinystories |
|
|
| A minimal character-level GPT model trained from scratch on the **tinystories** dataset. |
|
|
| ## Model Details |
|
|
| | Property | Value | |
| |----------|-------| |
| | Architecture | MicroGPT (decoder-only transformer) | |
| | Parameters | 419,712 | |
| | Vocab Size | 74 (character-level) | |
| | Block Size | 128 tokens | |
| | Layers | 2 | |
| | Attention Heads | 4 | |
| | Embedding Dim | 128 | |
| | Training Steps | 500 | |
| | Best Val Loss | 2.2494213581085205 | |
| | Dataset | tinystories | |
|
|
| ## Quick Start |
|
|
| ```bash |
| # Clone the repo |
| git clone https://huggingface.co/{{cookiecutter.repo_id if cookiecutter else 'your-username/micro-gpt-' + config['dataset']}} |
| cd micro-gpt-tinystories |
| |
| # Install dependencies |
| pip install -r requirements.txt |
| |
| # Generate text |
| python inference.py --prompt "Once upon a time" |
| ``` |
|
|
| ## Usage in Python |
|
|
| ```python |
| import torch |
| from models.micro_gpt import MicroGPT |
| |
| # Load config |
| import json |
| with open('config.json') as f: |
| config = json.load(f) |
| |
| # Build model |
| model = MicroGPT( |
| vocab_size=config['vocab_size'], |
| block_size=config['block_size'], |
| n_layer=config['n_layer'], |
| n_head=config['n_head'], |
| n_embd=config['n_embd'], |
| dropout=config['dropout'], |
| ) |
| model.load_state_dict(torch.load('pytorch_model.bin', map_location='cpu')) |
| model.eval() |
| |
| # Load tokenizer |
| with open('tokenizer.json') as f: |
| tokenizer = json.load(f) |
| |
| # Encode prompt |
| prompt = "Once upon a time" |
| indices = [tokenizer['stoi'].get(c, 0) for c in prompt] |
| input_ids = torch.tensor([indices], dtype=torch.long) |
| |
| # Generate |
| output_ids = model.generate(input_ids, max_new_tokens=200, temperature=0.9, top_k=40) |
| text = ''.join(tokenizer['itos'][str(i)] for i in output_ids[0].tolist()) |
| print(text) |
| ``` |
|
|
| ## Training |
|
|
| This model was trained using the project's training pipeline: |
|
|
| ```bash |
| python run_model.py --train_model --arch micro_glm --dataset tinystories --max_steps 500 |
| ``` |
|
|
| ## Files in this repository |
|
|
| | File | Description | |
| |------|-------------| |
| | `models/` | Full model source code (MicroGPT architecture) | |
| | `pytorch_model.bin` | Trained model weights | |
| | `config.json` | Model hyperparameters | |
| | `tokenizer.json` | Character-level tokenizer | |
| | `inference.py` | Ready-to-use inference script | |
| | `requirements.txt` | Python dependencies | |
|
|
