Commit History
-
Regenerate tokenizer.tkz as format v13 (self-contained: added/special tokens stored in-file)
Verified signatureb43f2c5 -
v12 tkz: exact precompiled charsmap + f64 unigram scores
Verified signature28eeab4
-
Add model card README
Verified signatured9529cc -
Add tokenizer.json from google-t5/t5-base
Verified signature008f8a1 -
Add tokie banner
Verified signature55edcc0 -
Update tokenizer.tkz to v11 (with pad_token_id)
Verified signature1f55e2d -
initial commit
Verified signature8922d75