---
license: mit
library_name: pytorch
tags:
- nanogpt
- tinystories
- gpt2
datasets:
- roneneldan/TinyStories
---
# KekGPT

A ~30M-parameter GPT trained from scratch on TinyStories with unmodified [nanoGPT](https://github.com/karpathy/nanoGPT) (MIT, Andrej Karpathy). Trained on Kaggle.

- Config: 6 layers, 6 heads, n_embd 384, block_size 256, no bias, dropout 0.0, vocab 50304 (GPT-2 BPE 50257 padded to 50304)
- Params: 30,036,864 total (tied embeddings); 29,938,560 excluding positional embeddings, which is the 29.94M nanoGPT prints
- Checkpoint: `KekGPT/nanoGPT/out-kekgpt/ckpt.pt` (iter 8000, best val loss 1.4365, 360,488,092 bytes), exact file from training, not modified
- Tokenizer: tiktoken `gpt2`
- `KekGPT.zip` has the same folder in one file
- `KekGPT/kekgpt_ui_cell.py`: the notebook cell for the top-10 next-token UI (Flask + optional Cloudflare quick tunnel). Softmax is over the full 50304 logits with no masking or renormalizing; ids >= 50257 show as "[unused id N]". It runs on CPU (about 0.09-0.28 s per call, measured on a 2-thread Kaggle CPU). Edit the `s=` path to your nanoGPT folder.
- `KekGPT/samples.txt`: sample stories from the checkpoint
- nanoGPT source files are included unmodified under `KekGPT/nanoGPT/` with their MIT license. Training data is TinyStories (CDLA-Sharing-1.0), not included.

Small model trained on children's stories: it writes simple fiction and is not a general assistant.