Instructions to use EmnaBou/TD-tokenizer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use EmnaBou/TD-tokenizer with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("token-classification", model="EmnaBou/TD-tokenizer")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForTokenClassification tokenizer = AutoTokenizer.from_pretrained("EmnaBou/TD-tokenizer") model = AutoModelForTokenClassification.from_pretrained("EmnaBou/TD-tokenizer", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update config.json
Browse files- config.json +1 -1
config.json
CHANGED
|
@@ -13,5 +13,5 @@
|
|
| 13 |
"tokenize_chinese_chars": true,
|
| 14 |
"tokenizer_class": "BertTokenizer",
|
| 15 |
"unk_token": "[UNK]",
|
| 16 |
-
"model_type": "
|
| 17 |
}
|
|
|
|
| 13 |
"tokenize_chinese_chars": true,
|
| 14 |
"tokenizer_class": "BertTokenizer",
|
| 15 |
"unk_token": "[UNK]",
|
| 16 |
+
"model_type": "Bert"
|
| 17 |
}
|