Commit History
Yesterday
-
Release Nep-Tokenizer v2 (64k): Native Hugging Face BPE with Mark-Aware Pre-tokenization and 100% Devanagari Block Coverage
- Architecture: Native Rust BPE (PreTrainedTokenizerFast) - Pre-tokenization: Mark-Aware Regex [\p{L}\p{M}]+ & Digit Grouping [0-9०-९]{1,3} - Full 17-Model Benchmark Evaluation and Publication Figures - Affiliation: DOAI & ILPRL, Kathmandu Universitya4c3ef3
