model card overhaul

README.md

@@ -1,64 +1,130 @@
 ---
+language:
+- en
 license: apache-2.0
+tags:
+- text-generation
+- rnn
+- gru
+- char-level
+- ascii
+- binary
+- tiny-stories
+- pytorch
 datasets:
 - roneneldan/TinyStories
-language:
-- en
 pipeline_tag: text-generation
-tags:
-- ASCII
-- BINARY
-- RNN
 ---
 
-Built ran and trained within 48hrs. - Inprogress By @ContextReq
-
-
-A tiny char-level language model that learns only the binary u8 id stream; all "english/symbols" rendering happens at runtime via a decoder that maps tags (<spc>, <nwl>, <bos>…) to their effects and drops them from display.
-  Status: As of this being writen, a full run has begun.
-
-Architecture (modular, 12-layer naming, 4 building blocks)
-layer	component	                       config
-0       binary input (memmap, chunk/tapes) B=32, T=1024
-1	    embedding wte + wpe	               vocab=109, d=512
-2–3	    GRU blocks (pre-LN + residual) ×2  d=512
-11	    head (ln_f + lm_head, tied to wte) 109-way
-
-3,735,040 params, fp32 ≈ 15MB weights.
-Causal by construction; hidden state carried across windows.
-   
-Data
-  TinyStoriesV2-GPT4: 2,118,989 stories → u8 id stream, 1,799,636,685 train ids / 95,696,324 val (95/5, story-boundary split, byte-exact verified).
-  Vocab: 109 tokens — 8 specials, 94 ASCII, 7 smart punct; vocab.txt == vocab.bin asserted at every train launch.
-
-Training
-  Stateful truncated BPTT (k1 = k2 = 1024): gradients truncated per window, state flows forward, reset only at sweep end — truncation + statefulness + tape continuity formally proven in --check.
-  AdamW, lr 4e-4, warmup 500, cosine → 4e-5, grad-clip 1.0.
-  Hardware: GTX 1660 SUPER (6GB, Turing) — ~95k tok/s, ~5.5h/epoch; dataset stays on CPU memmap (measured: moving it to GPU = +0.08%).
-  Checkpoints every 1,000 steps: step, train/val loss, lr, sweep position, weights, plus 2 sampled stories — resumable.
-
-used complete.py to sample completions of a choosen checkpoint.  
-  
-As of the first checkpoint: 
-epoch 0 step 1000: loss 0.910 | 104,813 tok/s | lr 4.00e-04
-val loss @ step 1000: 0.930
---- sample 0 (257 ids) ---
-Once upon a time, there was a little big park. All the other key and the little girl was so glad to help with her friends for the world on his sweet. She watched their testination. She smiled and said, "Its to make you better. Ben asks."
-"Sorry, we can go
---- sample 1 (257 ids) ---
-Once upon a time, there was a little girl named Ben. She didn't know whatch that made the store enveloes was not safe. He looked up and saw that he could hold her on the forest and jumped, on it again. The wind was very fast and he also could not want to l
-
-val loss @ step 5000: 0.768
---- sample 0 (257 ids) ---
-Once there was a big tree in Christ. He loved to play with his toys. He loved to help his family about something fun.
-One day, while they were still three years old, having lots of fun. But then, a little boy named Tom went to the pall and took the bowl aw
---- sample 1 (257 ids) ---
-Once there was a small town, so she looked out at the breeze. She was so happy to have made a regular ball and flew through his thermicale.
-The boy stood so beautifully! He flew too and found things like fluffy. He had the best day ever!
-Jimmy's mom ate al
-
-val loss @ step 10000: 0.719
---- sample 0 (257 ids) ---
-Once upon a time there was a small birthday who loved to shiver. He loved to flash on the other birds for the navy boxes and leave his paws to solve things. One day, his dad came to listen. He saw his friends at the shelter and started to smile. Suddenly,
---- sample 1 (257 ids) ---
-Once upon a time, there was a big storm. It was so soft and right to leaning a safe place. What was hot outside? The doctor wore it all day long. One day, the furries were gone. He looked around, but still soon fell asleep. Big gray boat left him a orange
+# Notio-3.7M-RNN-v1
+
+A deliberately minimal **char-level language model** (3.7M params, 2-layer GRU) trained on TinyStories.
+Notio's core idea: the model learns **only the binary u8 token-id stream** — all human-readable
+"english/symbols" rendering happens at runtime through a decoder that maps special tags
+(`<spc>`, `<nwl>`, `<bos>`, `<eos>`...) to their effects and drops them from display.
+
+> **Status:** v1 training in progress (27,000-step cosine run on a single GTX 1660 SUPER).
+> Milestone checkpoints uploaded: `notio1ksteps.pt`, `notio5k.pt`, `notio10k.pt`.
+> Card reflects step ~11,000 and will be updated at completion.
+
+## Design
+
+- **Vocabulary:** 109 tokens — 8 specials + 94 printable ASCII + 7 smart punctuation.
+  Every token id fits in **one byte**, so the entire corpus is a plain u8 stream.
+- **Boundary tags:** each story is wrapped `<bos> … <nwl><eos>`; whitespace inside the stream
+  is explicit (`<spc>`, `<nwl>`, `<tab>`).
+- **Runtime decoder:** tags map to effects at decode time (`<nwl>`→newline, `<spc>`→space,
+  `<bos>/<eos>/<pad>/<msk>`→dropped). Tags are data in the stream, never shown to humans.
+
+## Architecture
+
+| component | detail |
+|---|---|
+| embedding | wte (109×512) + wpe (1024×512) |
+| core | 2 × GRU blocks, d_model=512, pre-LayerNorm + residual |
+| head | LayerNorm + lm_head (109-way), **weights tied to wte** |
+| total params | **3,735,040** (fp32, ~15 MB weights) |
+| context | block_size 1024, **stateful truncated BPTT** (k1 = k2 = 1024) |
+
+The stack is modular: layer 0 (data) → layer 1 (embedding) → 2× layer 2 (GRU block) → layer 11 (head).
+A transformer-block variant exists in the repo and can be swapped in without touching other layers.
+
+## Data
+
+- **roneneldan/TinyStories** (TinyStoriesV2-GPT4-train), cleaned and filtered to stories
+  containing only in-vocab characters: **2,118,989 stories**.
+- Char-level u8 id stream: **1,799,636,685 train ids / 95,696,324 val ids** (95/5,
+  split on a story boundary — train ends `<eos>`, val starts `<bos>`, concat hash-verified).
+- Pipeline: tag → purge → tokenize; every stage parallelized and byte-exact verified
+  (sha256 manifest, tokenizer round-trip `ids identical: True`, `display identical: True`).
+
+## Training
+
+| hyperparameter | value |
+|---|---|
+| optimizer | AdamW, grad-clip 1.0 |
+| schedule | lr 4e-4, warmup 500 steps, cosine → 4e-5 over 27k steps |
+| batch | B=32, T=1024 (32,768 tokens/step) |
+| BPTT | gradients truncated per 1024-window, RNN state carried & detached, reset only at sweep end |
+| hardware | single GTX 1660 SUPER (6 GB), ~105k tok/s, ~2.3 h/run |
+| checkpoints | every 1,000 steps: train/val loss, lr, weights, + 2 sampled stories (resumable) |
+
+### Training progress
+
+| step | train loss | val loss | sample quality |
+|---|---|---|---|
+| 1,000 | 0.910 | 0.930 | mostly-correct words, grammar errors |
+| 3,000 | 0.756 | 0.803 | real story skeleton, semantic drift |
+| 5,000 | 0.692 | 0.768 | plot beats + dialogue, entity drift |
+| 7,000 | 0.708 | 0.740 | vocabulary expanding, still dreamy |
+| 10,000 | 0.683 | 0.719 | clean grammar, occasional noun swaps |
+| 11,000 | ~0.70 | **0.714** | **first story holding its protagonist end-to-end** |
+
+## Sample output
+
+Step 11,000, temperature 0.9, prompted with `<bos>`:
+
+> Once upon a time, there was a young boy named Timmy. Timmy loved to play outside with his
+> little bird. One day, Timmy started to rain, and his doggy came by. He waited until his mommy
+> finally back away and they became friends. They had fun playing together
+
+(Remaining errors at this stage: occasional subject swaps — "Timmy started to rain" — and
+invented words. Generation stops at `<eos>` once the model emits it.)
+
+## Usage
+
+Checkpoints embed the model config (load with `weights_only=False`). Repo tools:
+`src/sampler.py` (load/generate/decode) and `src/complete.py` (CLI).
+
+```bash
+# CLI: prompt in natural text, N completions, temperature/top-k control
+python3 complete.py "Once upon a time, there was" \
+    --checkpoint notio10k.pt --n 3 --temp 0.9 --top-k 40 --max-tokens 256
+```
+
+```python
+import sys
+sys.path.insert(0, "path/to/repo")          # repo root: notio.py, complete.py, sampler.py
+from sampler import load_model, encode_prompt, generate, ids_to_display
+
+model, device, ck = load_model("notio10k.pt")
+ids = generate(model, device, encode_prompt("Once upon a time"),
+               max_tokens=256, temperature=0.9, top_k=0)
+print(ids_to_display(ids))   # human text; boundary tags dropped/mapped
+```
+
+## Verification scoreboard
+
+- Data files: sha256 manifest check (all OK)
+- train/val split: concat hash == source hash, boundary-aligned
+- Tokenizer: full-corpus round-trip byte-exact (ids + display)
+- BPTT: truncation (detached state is a graph leaf), statefulness, and tape continuity formally asserted
+- GRU vs LSTM bake-off at equal settings: LSTM −1.5% loss, GRU +14% throughput → GRU chosen
+
+## Limitations
+
+- Char-level: occasional misspellings and invented words (e.g. "spaceshimane")
+- Small capacity: dream-logic entity swaps persist at this scale ("a small birthday who loved to shiver")
+- Occasionally repeats loops near low temperature; sample with temperature/top-k
+- Trained on synthetic GPT-4-generated children's stories — style is narrow and young-reader oriented
+- English only, 109-token fixed vocabulary