Commit History
-
working tokenizer
Verified signaturef9d6e33
-
Upload 6 files
Verified signatureec0e62c
-
simpler tokenizer
Verified signatureca2f1bc
-
smallest tokenizer and correct token indices
Verified signature3af8b1c -
phi tokenizer
Verified signature42e1b9e -
fast llama tokenizer
Verified signature835dfcd -
tokenizer minimized
Verified signature37a9b8c -
llama tokenizer
Verified signature0c7047e -
hidden_size=8
Verified signature7b56ddb
-
dummy-phi3 recreated
Verified signature6f5df4c -
dummy-phi3 first try
Verified signature234f2b0