codebert-llvm-ic-v0 / tokenizer_config.json
iyaja's picture
add tokenizer
92cad50
Raw
History Blame Contribute Delete
78 Bytes
{"max_len_single_sentence": 512, "tokenizer_class": "PreTrainedTokenizerFast"}