Token Classification
Transformers
Indonesian
tokenizer
bpe
bahasa-indonesia
indonesian
nlp
text-processing
subword-tokenization
Instructions to use romizone/bpe-tokenizer-id with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use romizone/bpe-tokenizer-id with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("token-classification", model="romizone/bpe-tokenizer-id")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("romizone/bpe-tokenizer-id", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Upload BPE Tokenizer Bahasa Indonesia
Browse files
README.md
CHANGED
|
@@ -91,6 +91,10 @@ IDs: [234, 567, 891, 123]
|
|
| 91 |
|
| 92 |
Built by **Jekardah AI Lab** 🇮🇩
|
| 93 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 94 |
## License
|
| 95 |
|
| 96 |
MIT License
|
|
|
|
| 91 |
|
| 92 |
Built by **Jekardah AI Lab** 🇮🇩
|
| 93 |
|
| 94 |
+
- **Email:** rominur@gmail.com
|
| 95 |
+
- **Website:** [rominur.com](https://rominur.com)
|
| 96 |
+
- **Lab:** [Jekardah.com](https://jekardah.com)
|
| 97 |
+
|
| 98 |
## License
|
| 99 |
|
| 100 |
MIT License
|