Commit History
-
Update README.md
Verified signaturefrom transformers import T5Tokenizer text = "сайн уу" mn_tokenizer = T5Tokenizer(vocab_file="mn_tokenizer.model") tokens = mn_tokenizer.tokenize(text) encoded = mn_tokenizer.encode(text) decoded = mn_tokenizer.decode(encoded) print("Original:", text) print("Mongolian tokenizer tokens:", tokens) print("Decoded :", decoded)a665cb6 -
Create README.md
Verified signatureea9726c -
Upload folder using huggingface_hub
Verified signature6b244ee -
initial commit
Verified signature3ddffbd