Instructions to use DanielHesslow/byte_level_tokenizer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use DanielHesslow/byte_level_tokenizer with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("DanielHesslow/byte_level_tokenizer", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update tokenizer.json
Browse files- tokenizer.json +1 -1
tokenizer.json
CHANGED
|
@@ -1124,7 +1124,7 @@
|
|
| 1124 |
[
|
| 1125 |
"每",
|
| 1126 |
0.0
|
| 1127 |
-
]
|
| 1128 |
],
|
| 1129 |
"byte_fallback": false
|
| 1130 |
}
|
|
|
|
| 1124 |
[
|
| 1125 |
"每",
|
| 1126 |
0.0
|
| 1127 |
+
]
|
| 1128 |
],
|
| 1129 |
"byte_fallback": false
|
| 1130 |
}
|