add tokenizer

Files changed (1) hide show

vocab.json CHANGED Viewed

	@@ -1 +1 @@
1	- {"ق": 1, "و": 2, "پ": 3, "ہ": 4, "غ": 5, "ڈ": 6, "ل": 7, "ٹ": 8, "گ": 9, "ر": ~~10, "ف":~~ 11, "ی": 12, "ت": 13, "ک": 14, "ے": 15, "ش": 16, "ں": 17, "ھ": 18, "س": 19, "م": 20, "آ": 21, "ا": 22, "د": 23, "ن": 24, "ح": 25, "\|": 0, "[UNK]": 26, "[PAD]": 27}


1	+ {"ح": 0, "آ": 1, "ف": 2, "م": 3, "ھ": 4, "ہ": 5, "گ": 6, "ق": 7, "پ": 8, "د": 9, "ت": 11, "غ": 12, "ں": 13, "و": 14, "ل": 15, "ے": 16, "ک": 17, "ر": 18, "س": 19, "ش": 20, "ٹ": 21, "ی": 22, "ا": 23, "ڈ": 24, "ن": 25, "\|": 10, "[UNK]": 26, "[PAD]": 27}