Commit History
-
Upload tokenizer
Fixes #16, which showed that one of the merges is missing, which creates different outputs. The issue with this is that finetuned models were trained using this.2ca60a1
-
add special tokens for fast (#25)
- add special tokens for fast (55f5fcdfb66210824046aeee3c5c1028bc364427)013fe3b
-
Upload tokenizer.json (#11)
- Upload tokenizer.json (a216e1ddde75fbe924e49e58f870d90476fe724b)74356c9
-
Changes for fast tokenizer (#7)
- Update tokens (cbad20c57e9c29a3ac3eefa2ee23e88c69f96bbf) - Add tokenizer.json (7527b64ee55cf354da2bd36a871313974cd36684) Co-authored-by: Jonatan Kłosko <jonatanklosko@users.noreply.huggingface.co>5815f0e
