col-minilm / tokenizer.json

Commit History

  1. clean up tokenizer and add [unused0] and [unused1] as special tokens

    Jo Kristian Bergum
    85b42e5

  1. Upload tokenizer.json

    4eafbac