ContextReq commited on
Commit
970cc36
·
verified ·
1 Parent(s): 44e85e7

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +9 -20
README.md CHANGED
@@ -49,7 +49,6 @@ Notio's core idea: the model learns **only the binary u8 token-id stream** — a
49
  | context | block_size 1024, **stateful truncated BPTT** (k1 = k2 = 1024) |
50
 
51
  The stack is modular: layer 0 (data) → layer 1 (embedding) → 2× layer 2 (GRU block) → layer 11 (head).
52
- A transformer-block variant exists in the repo and can be swapped in without touching other layers.
53
 
54
  ## Data
55
 
@@ -71,27 +70,17 @@ A transformer-block variant exists in the repo and can be swapped in without tou
71
  | hardware | single GTX 1660 SUPER (6 GB), ~105k tok/s, ~2.3 h/run |
72
  | checkpoints | every 1,000 steps: train/val loss, lr, weights, + 2 sampled stories (resumable) |
73
 
74
- ### Training progress
75
-
76
- | step | train loss | val loss | sample quality |
77
- |---|---|---|---|
78
- | 1,000 | 0.910 | 0.930 | mostly-correct words, grammar errors |
79
- | 3,000 | 0.756 | 0.803 | real story skeleton, semantic drift |
80
- | 5,000 | 0.692 | 0.768 | plot beats + dialogue, entity drift |
81
- | 7,000 | 0.708 | 0.740 | vocabulary expanding, still dreamy |
82
- | 10,000 | 0.683 | 0.719 | clean grammar, occasional noun swaps |
83
- | 11,000 | ~0.70 | **0.714** | **first story holding its protagonist end-to-end** |
84
-
85
  ## Sample output
86
 
87
- Step 11,000, temperature 0.9, prompted with `<bos>`:
88
-
89
- > Once upon a time, there was a young boy named Timmy. Timmy loved to play outside with his
90
- > little bird. One day, Timmy started to rain, and his doggy came by. He waited until his mommy
91
- > finally back away and they became friends. They had fun playing together
92
-
93
- (Remaining errors at this stage: occasional subject swaps — "Timmy started to rain" — and
94
- invented words. Generation stops at `<eos>` once the model emits it.)
 
95
 
96
  ## Usage
97
 
 
49
  | context | block_size 1024, **stateful truncated BPTT** (k1 = k2 = 1024) |
50
 
51
  The stack is modular: layer 0 (data) → layer 1 (embedding) → 2× layer 2 (GRU block) → layer 11 (head).
 
52
 
53
  ## Data
54
 
 
70
  | hardware | single GTX 1660 SUPER (6 GB), ~105k tok/s, ~2.3 h/run |
71
  | checkpoints | every 1,000 steps: train/val loss, lr, weights, + 2 sampled stories (resumable) |
72
 
 
 
 
 
 
 
 
 
 
 
 
73
  ## Sample output
74
 
75
+ As of; val loss @ step 15000: 0.700
76
+ --- sample 0 (257 ids) ---
77
+ Once upon a time, there was a boy named of a coat, a big house. Mark's mom would put on his bike and determined to order more things. So, his mom asked her if he said to her, "Can you teach me the worker today?"
78
+ "Well, they worked hard, pushing and making
79
+ --- sample 1 (257 ids) ---
80
+ Once upon a time, there was a princess named Anna. She loved to share her toys when she was determined. One day, she was red and brave and found some black boat.
81
+ Suddenly, she heard a voice. It was a quiet.
82
+ "Shh, set, geeth in the water!" said True.
83
+ Lola
84
 
85
  ## Usage
86