Hugging Face's logo

cbrownpinilla
/
bottom-3B

PyTorch
hf_olmo
custom_code
Model card Files
xet
 Community
bottom-3B / tokenization_olmo_fast.py
cbrownpinilla's picture
cbrownpinilla
HF integration
5fc231f
Raw History Blame Contribute Delete
187 Bytes
from hf_olmo.tokenization_olmo_fast import OLMoTokenizerFast, OLMoConfig
from transformers import AutoTokenizer
AutoTokenizer.register(OLMoConfig, fast_tokenizer_class=OLMoTokenizerFast)