token detail update
Browse files
README.md
CHANGED
|
@@ -9,4 +9,6 @@ pipeline_tag: text-generation
|
|
| 9 |
|
| 10 |
>:), keep an eye out or don't, who knows - im insane muhehehe.
|
| 11 |
|
| 12 |
-
|
|
|
|
|
|
|
|
|
| 9 |
|
| 10 |
>:), keep an eye out or don't, who knows - im insane muhehehe.
|
| 11 |
|
| 12 |
+
ill clean this page up at some point.
|
| 13 |
+
|
| 14 |
+
109 token IDS for vocab.txt/.bin, line number is implied token id, token value e.g. line 99 == id 99, "token", coverted into binary(.bin).
|