Update README.md
Browse files
README.md
CHANGED
|
@@ -49,7 +49,6 @@ Notio's core idea: the model learns **only the binary u8 token-id stream** — a
|
|
| 49 |
| context | block_size 1024, **stateful truncated BPTT** (k1 = k2 = 1024) |
|
| 50 |
|
| 51 |
The stack is modular: layer 0 (data) → layer 1 (embedding) → 2× layer 2 (GRU block) → layer 11 (head).
|
| 52 |
-
A transformer-block variant exists in the repo and can be swapped in without touching other layers.
|
| 53 |
|
| 54 |
## Data
|
| 55 |
|
|
@@ -71,27 +70,17 @@ A transformer-block variant exists in the repo and can be swapped in without tou
|
|
| 71 |
| hardware | single GTX 1660 SUPER (6 GB), ~105k tok/s, ~2.3 h/run |
|
| 72 |
| checkpoints | every 1,000 steps: train/val loss, lr, weights, + 2 sampled stories (resumable) |
|
| 73 |
|
| 74 |
-
### Training progress
|
| 75 |
-
|
| 76 |
-
| step | train loss | val loss | sample quality |
|
| 77 |
-
|---|---|---|---|
|
| 78 |
-
| 1,000 | 0.910 | 0.930 | mostly-correct words, grammar errors |
|
| 79 |
-
| 3,000 | 0.756 | 0.803 | real story skeleton, semantic drift |
|
| 80 |
-
| 5,000 | 0.692 | 0.768 | plot beats + dialogue, entity drift |
|
| 81 |
-
| 7,000 | 0.708 | 0.740 | vocabulary expanding, still dreamy |
|
| 82 |
-
| 10,000 | 0.683 | 0.719 | clean grammar, occasional noun swaps |
|
| 83 |
-
| 11,000 | ~0.70 | **0.714** | **first story holding its protagonist end-to-end** |
|
| 84 |
-
|
| 85 |
## Sample output
|
| 86 |
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
|
|
|
| 95 |
|
| 96 |
## Usage
|
| 97 |
|
|
|
|
| 49 |
| context | block_size 1024, **stateful truncated BPTT** (k1 = k2 = 1024) |
|
| 50 |
|
| 51 |
The stack is modular: layer 0 (data) → layer 1 (embedding) → 2× layer 2 (GRU block) → layer 11 (head).
|
|
|
|
| 52 |
|
| 53 |
## Data
|
| 54 |
|
|
|
|
| 70 |
| hardware | single GTX 1660 SUPER (6 GB), ~105k tok/s, ~2.3 h/run |
|
| 71 |
| checkpoints | every 1,000 steps: train/val loss, lr, weights, + 2 sampled stories (resumable) |
|
| 72 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
## Sample output
|
| 74 |
|
| 75 |
+
As of; val loss @ step 15000: 0.700
|
| 76 |
+
--- sample 0 (257 ids) ---
|
| 77 |
+
Once upon a time, there was a boy named of a coat, a big house. Mark's mom would put on his bike and determined to order more things. So, his mom asked her if he said to her, "Can you teach me the worker today?"
|
| 78 |
+
"Well, they worked hard, pushing and making
|
| 79 |
+
--- sample 1 (257 ids) ---
|
| 80 |
+
Once upon a time, there was a princess named Anna. She loved to share her toys when she was determined. One day, she was red and brave and found some black boat.
|
| 81 |
+
Suddenly, she heard a voice. It was a quiet.
|
| 82 |
+
"Shh, set, geeth in the water!" said True.
|
| 83 |
+
Lola
|
| 84 |
|
| 85 |
## Usage
|
| 86 |
|