Commit History
-
working tokenizer
Verified signaturef9d6e33
-
Upload 6 files
Verified signatureec0e62c
-
simpler tokenizer
Verified signatureca2f1bc -
max length to over 1,000,000
Verified signatureb15d526 -
tokenizer chat template
Verified signature6e50a01
-
smallest tokenizer and correct token indices
Verified signature3af8b1c -
phi tokenizer
Verified signature42e1b9e -
fast llama tokenizer
Verified signature835dfcd -
tokenizer minimized
Verified signature37a9b8c -
llama tokenizer
Verified signature0c7047e -
byt5 tokenizer
Verified signaturebe43126
-
dummy-phi3 recreated
Verified signature6f5df4c -
dummy-phi3 first try
Verified signature234f2b0