konko
/

bengali-bpe-tokenizer

Token Classification

Model card Files Files and versions

bengali-bpe-tokenizer / tokenizer.json

konko's picture

v0.2: 64k literary-weighted tokenizer, beats IndicBERTv2 on every held-out register

3ba25d0 verified 3 months ago

Raw History Contribute Delete

4.76 MB

File too large to display, you can check the raw version instead.