| --- |
| language: en |
| license: apache-2.0 |
| tags: |
| - tiny |
| - slm |
| - small-language-model |
| - from-scratch |
| - llama |
| - rope |
| - swiglu |
| - rmsnorm |
| - text-generation |
| - tinystories |
| pipeline_tag: text-generation |
| library_name: transformers |
| --- |
| |
| # StoryLM-10M |
|
|
| A ~12.6M parameter LLaMA-style causal language model trained from scratch on TinyStories. |
|
|
| ## Architecture |
|
|
| | Parameter | Value | |
| |-----------|-------| |
| | d_model | 256 | |
| | n_heads | 8 (MHA) | |
| | n_kv_heads | 8 | |
| | n_layers | 8 | |
| | head_dim | 32 | |
| | FFN dim | 1024 (SwiGLU, 4x) | |
| | Vocab size | 8192 | |
| | Context length | 512 | |
| | Tied embeddings | Yes | |
| | Total params | 12,603,648 | |
|
|
| ## Training |
|
|
| - **Data**: TinyStories (2,119,719 stories, 533.7M tokens) |
| - **Steps**: 24,400 (batch 4, seq 512) |
| - **Optimizer**: AdamW, LR 3e-4, cosine schedule, 500-step warmup |
| - **Best val loss**: 1.8635 |
| - **Val perplexity**: 6.45 |
|
|
| ## Generation Samples |
|
|
| ``` |
| Prompt: Once upon a time, there was a little |
| Output: Once upon a time, there was a little |
| |
| Prompt: The cat sat on the |
| Output: The cat sat on the |
| |
| Prompt: In the beginning, the world was |
| Output: In the beginning, the world was |
| |
| Prompt: A small robot named |
| Output: A small ro bot named |
| |
| Prompt: Every morning, the sun |
| Output: Every morning, the sun |
| ``` |
|
|
| ## Notes |
|
|
| - Trained on a single RTX 5090 (32 GB) in ~281 seconds of training time. |
| - The model is a custom implementation (not HuggingFace transformers-compatible out of the box). |
| - Weights are stored as a single PyTorch checkpoint (`model.pt`). |
