Hugging Face's logo

SurendraVB
/
Synaptic-Edge-Tokenizer-65K

Text Generation
PyTorch
tokenizer
bpe
byte-level
code
stem
neuromorphic
synaptic-edge
triton
Model card Files
xet
 Community
Synaptic-Edge-Tokenizer-65K / tokenizer_config.json

Commit History

fix: export standard tokenizer_config.json for AutoTokenizer
0ecce52
verified

SurendraVB commited on

feat: add tokenizer_config.json for AutoTokenizer support
17e2908
verified

SurendraVB commited on