Upload model files

README.md

@@ -0,0 +1,62 @@
+---
+license: mit
+library_name: pytorch
+tags:
+  - text-generation
+  - gpt
+  - from-scratch
+  - tinystories
+datasets:
+  - roneneldan/TinyStories
+language:
+  - en
+metrics:
+  - perplexity
+---
+
+# Tiny GPT (5.8M parameters)
+
+A small GPT-2-style decoder-only transformer, written from scratch in PyTorch (no `transformers` classes), trained on a
+laptop CPU on a subset of [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories). It continues short children's stories.
+Code, training logs and the full write-up are in the GitHub repository.
+
+## Model details
+| | |
+|---|---|
+| Architecture | pre-norm GPT, 6 layers, d_model 256, 8 heads, context 256, learned positions, tied embeddings |
+| Parameters | 5,853,184 |
+| Tokenizer | byte-level BPE, vocabulary 4096, trained on the training split only (`tokenizer.json`) |
+| Files | `model.safetensors`, `model_config.json` (our `ModelConfig`), `tokenizer.json`, `model_info.json` |
+
+## Training
+- **Data:** first 300,000 stories (60.8M tokens) of TinyStories V2 (GPT-4 generated); validation on 5,000 held-out stories.
+- **Run:** 9,500 steps x 8,192 tokens = 77.8M tokens (about 1.28 passes over the data), AdamW, peak lr 1e-3 with warmup and cosine decay, 12 CPU threads, 7.9 hours of running time on one laptop (Intel Core Ultra 9 185H).
+
+## Evaluation
+| Model | Validation loss | Perplexity |
+|---|---|---|
+| Uniform guess | 8.32 | 4096 |
+| Unigram counts | 5.88 | 357 |
+| Bigram counts | 3.65 | 38.4 |
+| **This model** | **1.688** | **5.41** |
+
+Perplexity is per BPE token with this 4096-token tokenizer, on TinyStories text, so it is not comparable to numbers from other tokenizers or datasets.
+
+## Intended use
+Learning, demos and experiments with small language models; generating simple children's-story-style text.
+
+## Limitations and risks
+- **Narrow domain.** It only knows the simple vocabulary and plots of TinyStories. It cannot follow instructions, answer questions or hold a conversation.
+- **Quality.** Stories are fluent at sentence level but repeat phrases, lose objects and plot threads, and sometimes contradict themselves. Samples in the repository's `docs/RESULTS.md` include failures, not just good examples.
+- **Synthetic data.** The training stories were written by a large language model, so the model inherits that model's stylistic and social biases (names, gender roles, settings). It has not been evaluated for bias or safety.
+- **Not for** factual content, advice, or anything where wrong or biased text could cause harm.
+
+## Data and licence
+Code and weights in this repository are released under the MIT licence. The training data, TinyStories by Ronen Eldan and Yuanzhi Li, is
+licensed CDLA-Sharing-1.0 on the dataset card; please credit it and check the dataset card for current terms.
+
+## Usage
+```bash
+git clone <github repo> && cd tiny-gpt && make setup
+TINYGPT_HF_REPO=<this repo id> TINYGPT_MODEL_DIR=./model uv run uvicorn tinygpt.serving.app:create_app --factory
+```

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6d9615059a7d499ad645f2f310fef70f91388a4804714becb008c72d90cb86ec
+size 23419560

model_config.json

@@ -0,0 +1,9 @@
+{
+  "vocab_size": 4096,
+  "context_length": 256,
+  "d_model": 256,
+  "n_layer": 6,
+  "n_head": 8,
+  "dropout": 0.1,
+  "attention_impl": "sdpa"
+}

model_info.json

@@ -0,0 +1,9 @@
+{
+  "run": "overnight_long",
+  "checkpoint": "best",
+  "training_steps": 9500,
+  "training_tokens": 77824000,
+  "val_loss": 1.6877,
+  "val_perplexity": 5.407,
+  "tokenizer_sha256": "f229f2e9cc79e0a4823e76f892ccabe9794bcd0ae7efc08a5ffbb57868a39229"
+}