README.md

metadata

license: mit
library_name: pytorch
tags:
  - nanogpt
  - tinystories
  - gpt2
datasets:
  - roneneldan/TinyStories

KekGPT

A ~30M-parameter GPT trained from scratch on TinyStories with unmodified nanoGPT (MIT, Andrej Karpathy). Trained on Kaggle.

• Config: 6 layers, 6 heads, n_embd 384, block_size 256, no bias, dropout 0.0, vocab 50304 (GPT-2 BPE 50257 padded to 50304)
• Params: 30,036,864 total (tied embeddings); 29,938,560 excluding positional embeddings, which is the 29.94M nanoGPT prints
• Checkpoint: KekGPT/nanoGPT/out-kekgpt/ckpt.pt (iter 8000, best val loss 1.4365, 360,488,092 bytes), exact file from training, not modified
• Tokenizer: tiktoken gpt2
• KekGPT.zip has the same folder in one file
• KekGPT/kekgpt_ui_cell.py: the notebook cell for the top-10 next-token UI (Flask + optional Cloudflare quick tunnel). Softmax is over the full 50304 logits with no masking or renormalizing; ids >= 50257 show as "[unused id N]". It runs on CPU (about 0.09-0.28 s per call, measured on a 2-thread Kaggle CPU). Edit the s= path to your nanoGPT folder.
• KekGPT/samples.txt: sample stories from the checkpoint
• nanoGPT source files are included unmodified under KekGPT/nanoGPT/ with their MIT license. Training data is TinyStories (CDLA-Sharing-1.0), not included.

Small model trained on children's stories: it writes simple fiction and is not a general assistant.