Hugging Face's logo

repetition-experiments
/
test

PyTorch
hf_olmo
Model card Files
xet
 Community
test / tokenization_olmo_fast.py
cbrownpinilla's picture
cbrownpinilla
Upload folder using huggingface_hub
d97faa2 verified
Raw History Blame Contribute Delete
187 Bytes
from hf_olmo.tokenization_olmo_fast import OLMoTokenizerFast, OLMoConfig
from transformers import AutoTokenizer
AutoTokenizer.register(OLMoConfig, fast_tokenizer_class=OLMoTokenizerFast)