Add model card, config, and loader for tinystories-50m

README.md

@@ -0,0 +1,55 @@
+---
+license: apache-2.0
+pipeline_tag: text-generation
+language: en
+tags:
+  - tiny
+  - tiny-lm
+  - tiny-model
+  - slm
+  - SLM
+  - small-language-model
+  - from-scratch
+  - tinystories
+metrics:
+  - perplexity
+  - val-loss
+---
+
+# tinystories-50m
+
+A **54,804,992-parameter** from-scratch BPE GPT trained on [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories). This is the scale-up of my [tinystories-24m](https://huggingface.co/Compactbot/tinystories-24m) — same narrow children's-story domain, ~2.3x the parameters.
+
+## What it is
+- **Architecture:** custom GPT — 16 layers, d=512, 8 heads, FFN 2048, seq 512, vocab 8192 (BPE). Weight-tied embeddings, RMSNorm, GELU, fused qkv. **Not** a stock transformers architecture — load it with the included `model_loader.py`.
+- **Data:** 447.9M TinyStories tokens, 1 full epoch (13,668 steps, batch 64, seq 512).
+- **Compute:** RTX 5090 (32 GB), ~8 steps/s, ~28 min.
+- **Params:** **54,804,992** exactly (verified against the safetensors header: 99 tensors, weight-tied so `tok.weight` is not double-counted).
+
+## Training
+- LR 3e-4, cosine decay, warmup 500. No divergence (unlike the 24m at 6e-4).
+- Best val loss **1.6566** @ step 13,250; final **1.6371** @ step 13,668.
+- In-domain perplexity (held-out TinyStories): **~5.24** (best ckpt).
+- ~8.2 tokens/param.
+
+## Quality (honest)
+Coherent, on-theme children's stories with proper punctuation and narrative structure. Sample (seed 0):
+
+> Once upon a time, there was a little girl named Lily. She loved to play in the garden with her dog, Buddy. One day, Lily's mom taught her a new game called wash. Lily was very excited and asked her mom if she could bring her favorite toy back. Lily went outside and filled a bucket with water. As she was catching the bucket, she noticed that it was wet and her clothes were wet. She quickly dried herself off and her clothes were all wet. She ran inside and asked her mom for help.
+
+Occasional artifacts are expected at this scale on a narrow domain: a garbled word here and there, pronoun slips, and short-range repetition. It is a **story generator**, not a general model — do not expect general English, reasoning, or instruction following.
+
+## What it is NOT good at
+- General-domain text (out of distribution; expect degradation).
+- Reasoning, math, multi-turn dialogue, instruction following.
+- Long coherence beyond ~140 generated tokens (trained at seq 512 but quality degrades with length).
+
+## Usage
+```python
+from model_loader import load, generate
+model, tok = load(".")          # needs torch, safetensors, tokenizers
+print(generate(model, tok, "Once upon a time, there was a little girl named Lily.", seed=0))
+```
+
+## Lineage
+- [tinystories-24m](https://huggingface.co/Compactbot/tinystories-24m) (24.59M, coherent at 18.2 tok/param) → **tinystories-50m** (54.80M, 8.2 tok/param).

config.json

@@ -0,0 +1,17 @@
+{
+  "architectures": ["CustomGPT"],
+  "model_type": "gpt",
+  "vocab_size": 8192,
+  "n_positions": 512,
+  "n_embd": 512,
+  "n_layer": 16,
+  "n_head": 8,
+  "n_inner": 2048,
+  "activation": "gelu",
+  "norm": "rmsnorm",
+  "tie_word_embeddings": true,
+  "attention": "causal-fused-qkv",
+  "n_params": 54804992,
+  "dtype": "float32",
+  "note": "Custom GPT (not a transformers architecture). Load with model_loader.py. Weights in model.safetensors (99 tensors, weight-tied: tok.weight serves both embedding and output projection)."
+}

model_loader.py

@@ -0,0 +1,62 @@
+"""Load and generate with Compactbot/tinystories-50m (custom GPT, 54.8M params).
+Requires: torch, safetensors, tokenizers. GPU optional (CPU works, slower).
+"""
+import torch, torch.nn as nn, torch.nn.functional as F
+from safetensors.torch import load_file
+from tokenizers import Tokenizer
+
+VOCAB,D,L,H,FFN,SEQ=8192,512,16,8,2048,512
+
+class RMSNorm(nn.Module):
+    def __init__(self,d): super().__init__(); self.w=nn.Parameter(torch.ones(d))
+    def forward(self,x): return self.w*x*torch.rsqrt(x.float().pow(2).mean(-1,keepdim=True)+1e-6)
+
+class Block(nn.Module):
+    def __init__(self,d,h):
+        super().__init__()
+        self.ln1=RMSNorm(d); self.ln2=RMSNorm(d)
+        self.qkv=nn.Linear(d,3*d,bias=False); self.proj=nn.Linear(d,d,bias=False)
+        self.fc1=nn.Linear(d,FFN,bias=False); self.fc2=nn.Linear(FFN,d,bias=False)
+        self.h,self.d=h,d
+    def forward(self,x):
+        B,T,Dd=x.shape; h=self.ln1(x)
+        qkv=self.qkv(h).view(B,T,3,self.h,Dd//self.h).transpose(2,1)
+        q,k,v=qkv[:,0].transpose(1,2),qkv[:,1].transpose(1,2),qkv[:,2].transpose(1,2)
+        att=F.scaled_dot_product_attention(q,k,v,is_causal=True)
+        x=x+self.proj(att.transpose(1,2).reshape(B,T,Dd))
+        return x+self.fc2(F.gelu(self.fc1(self.ln2(x))))
+
+class GPT(nn.Module):
+    def __init__(self):
+        super().__init__()
+        self.tok=nn.Embedding(VOCAB,D); self.pos=nn.Embedding(SEQ,D)
+        self.blocks=nn.ModuleList([Block(D,H) for _ in range(L)]); self.ln_f=RMSNorm(D)
+    def forward(self,x,targets=None):
+        h=self.tok(x)+self.pos(torch.arange(x.shape[1],device=x.device))
+        for b in self.blocks: h=b(h)
+        logits=self.ln_f(h)@self.tok.weight.t()
+        if targets is not None: return F.cross_entropy(logits.view(-1,VOCAB),targets.view(-1))
+        return logits
+
+def load(repo_dir=".",device="cuda" if torch.cuda.is_available() else "cpu"):
+    m=GPT().to(device); m.load_state_dict(load_file(f"{repo_dir}/model.safetensors"),strict=True)
+    m.eval()
+    return m,Tokenizer.from_file(f"{repo_dir}/tokenizer.json")
+
+def generate(model,tok,prompt,max_new=120,seed=0,temperature=0.8,device="cuda"):
+    g=torch.Generator(device=device).manual_seed(seed)
+    ids=torch.tensor([tok.encode(prompt).ids],device=device)
+    if ids.shape[1]>SEQ-4: ids=ids[:,-(SEQ-4):]
+    with torch.no_grad():
+        for _ in range(max_new):
+            with torch.autocast(device_type=device,dtype=torch.bfloat16):
+                logits=model(ids)
+            p=torch.softmax(logits[:,-1].float()/temperature,dim=-1)
+            nxt=torch.multinomial(p,1,generator=g); ids=torch.cat([ids,nxt],1)
+            if nxt.item()==1: break
+    return tok.decode(ids[0].tolist())
+
+if __name__=="__main__":
+    m,t=load(".")
+    print("params:",sum(p.numel() for p in m.parameters()))
+    print(generate(m,t,"Once upon a time, there was a little girl named Lily.",seed=0))