Hugging Face's logo

Spaces:
toksuite
/
tokenizer-comparison
Running

App Files Community
tokenizer-comparison

Commit History

  1. Update README.md Running

    gsaltintas
    Verified signature
    32f3caf

  2. Update README.md

    gsaltintas
    Verified signature
    21409c7

  1. Further improvements

    Gül Sena Altıntaş
    ce07484

  1. Improvements

    Gül Sena Altıntaş
    44cdae3

  1. Small improvement for visualization⁠

    Gül Sena Altıntaş
    199862a

  2. Added support for showing newlines⁠

    Gül Sena Altıntaş
    d9779a0

  1. Fixed tokenmonster issue

    Gül Sena Altıntaş
    f58b113

  1. Add normalization

    Gül Sena Altıntaş
    37a99cb

  2. subworsd mostly fixed, tokenmonster and byt5 still wrong [WIP]

    Gül Sena Altıntaş
    0c7d05e

  3. Added sample texts

    Gül Sena Altıntaş
    452a924

  4. Refactoring, and visual improvements

    Gül Sena Altıntaş
    c02e89e

  5. Added additional tokenizers

    Gül Sena Altıntaş
    3a08f05

  6. Create requirements.txt

    gsaltintas
    Verified signature
    14ea3d3

  7. Create app.py

    gsaltintas
    Verified signature
    6b9ea0b

  8. initial commit

    gsaltintas
    Verified signature
    b6070dc