Instructions to use mjbommar/ogbert-tokenizer-8192 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use mjbommar/ogbert-tokenizer-8192 with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("mjbommar/ogbert-tokenizer-8192", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Commit History
-
Upload OGBERT tokenizer (vocab_size=8192)
Verified signature3c76971
-
Upload OGBERT tokenizer (vocab_size=8192)
Verified signature1be5751
-
Add README with YAML metadata
Verified signature9c6a895 -
Add special_tokens_map
Verified signature0281a5f -
Add tokenizer_config with special tokens
Verified signature12a93c3
-
Upload folder using huggingface_hub
Verified signature5f87c08 -
initial commit
Verified signature76669df