Instructions to use omarmomen/babylm_tokenizer_32k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use omarmomen/babylm_tokenizer_32k with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("omarmomen/babylm_tokenizer_32k", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Commit History
-
Update README.md
Verified signature58792b1 -
Update README.md
Verified signature75fcf77 -
Update README.md
Verified signature2e80b2d -
Update README.md
Verified signature36d36a2 -
Create README.md
Verified signature52ec457
-
add tokenizer
eda0844 -
initial commit
0136203