markdown ate my token tags
Browse files
README.md
CHANGED
|
@@ -33,7 +33,7 @@ Notio's core idea: the model sees only token ids — a u8 stream, one byte per i
|
|
| 33 |
- **Boundary tags:** each story is wrapped `<bos> … <nwl><eos>`; whitespace inside the stream
|
| 34 |
is explicit (`<spc>`, `<nwl>`, `<tab>`).
|
| 35 |
- **Runtime decoder:** tags map to effects at decode time (`<nwl>`→newline, `<spc>`→space,
|
| 36 |
-
`<bos>
|
| 37 |
|
| 38 |
## Architecture
|
| 39 |
|
|
|
|
| 33 |
- **Boundary tags:** each story is wrapped `<bos> … <nwl><eos>`; whitespace inside the stream
|
| 34 |
is explicit (`<spc>`, `<nwl>`, `<tab>`).
|
| 35 |
- **Runtime decoder:** tags map to effects at decode time (`<nwl>`→newline, `<spc>`→space,
|
| 36 |
+
`<bos>` `<eos>` `<pad>` `<msk>`→dropped). Tags are data in the stream, never shown to humans.
|
| 37 |
|
| 38 |
## Architecture
|
| 39 |
|