Hugging Face's logo

mjbommar
/
ogbert-tokenizer-16384

Transformers
English
tokenizer
bpe
ogbert
modernbert
opengloss
Model card Files
xet
 Community
ogbert-tokenizer-16384

Commit History

  1. Upload OGBERT tokenizer (vocab_size=16384)

    mjbommar
    Verified signature
    f87e9d1

  2. Upload OGBERT tokenizer (vocab_size=16384)

    mjbommar
    Verified signature
    ae3b88b

  1. Add README with YAML metadata

    mjbommar
    Verified signature
    8a7f82c

  2. Add special_tokens_map

    mjbommar
    Verified signature
    3033174

  3. Add tokenizer_config with special tokens

    mjbommar
    Verified signature
    5c9cae5

  1. Upload folder using huggingface_hub

    mjbommar
    Verified signature
    9c3647e

  2. initial commit

    mjbommar
    Verified signature
    8a47d25