Hugging Face's logo

ByteSpanTokenisers
/
tokenizers

Model card Files
xet
 Community
tokenizers

Commit History

  1. Update frequency

    Zeb
    ad1ff6d

  2. Fix spaceless tokenizers

    Zeb
    d69e013

  3. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    b08c86b

  4. Fix spaceless tokenizers

    Zeb
    c58de58

  5. Remove ngram tokenizers

    Zeb
    7487a89

  6. Delete fw57M_Entropy_thresholdM_500

    codebyzeb
    Verified signature
    a802c9a

  7. Fix multilingual frequency

    Zeb
    d30eee2

  8. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    29dda54

  9. Delete fw57M_Entropy_thresholdM_500

    codebyzeb
    Verified signature
    b3b5b2d

  10. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    ad3d889

  11. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    7e118b6

  12. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    62a0567

  13. Fix tokenizers

    Zeb
    f9c3417

  14. Separate frequency tokenizers

    Zeb
    f25f2a5

  15. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    f9c85cd

  16. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    280544f

  17. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    3bd5966

  18. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    77097ea

  19. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    67b58c4

  20. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    f0fd6ae

  21. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    72c0e05

  22. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    114a45f

  23. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    06adb6a

  24. Upload folder using huggingface_hub

    codebyzeb
    1b719ca

  25. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    853f739

  1. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    60e26ce

  2. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    4763edc

  3. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    d2ac827

  4. Delete fw57M_Entropy_thresholdBX_600

    codebyzeb
    Verified signature
    a4fb558

  5. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    53333c9

  6. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    d000429

  7. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    0322999

  8. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    c404f26

  9. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    58beac7

  10. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    57b71d2

  11. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    d7e180f

  12. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    7c9b416

  1. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    77f8c2e

  2. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    ed6a20c

  3. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    006a6ab

  4. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    36edef8

  5. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    8d254aa

  6. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    420e742

  7. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    4a56766

  8. Remove faulty tokenizers

    Zeb
    faf7f42

  9. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    8994231

  10. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    c8158ef

  11. Rename and remove old tokenizers

    Zeb
    24ade47

  12. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    f554e97

  13. Upload folder using huggingface_hub

    codebyzeb
    Verified signature
    0f89be7