Update README.md
Browse files
README.md
CHANGED
|
@@ -55,7 +55,7 @@ language:
|
|
| 55 |
- mn
|
| 56 |
---
|
| 57 |
|
| 58 |
-
#
|
| 59 |
|
| 60 |
A 32,768-id byte-level Unigram tokenizer (HuggingFace `tokenizers` format, ByteLevel
|
| 61 |
pre-tokenizer and decoder) with 256 control tokens and 32,512 learned pieces. English-first,
|
|
|
|
| 55 |
- mn
|
| 56 |
---
|
| 57 |
|
| 58 |
+
# U32k: 32k byte-level Unigram tokenizer
|
| 59 |
|
| 60 |
A 32,768-id byte-level Unigram tokenizer (HuggingFace `tokenizers` format, ByteLevel
|
| 61 |
pre-tokenizer and decoder) with 256 control tokens and 32,512 learned pieces. English-first,
|