MyTranslation_300M_Base / tokenizer_config.json
freococo's picture
Release MyTranslation_300M_Base (298.76M inf / 9x RTX 5090, Val PPL 3.20 + custom_code, code & logs)
3bcdbab verified
Raw History Blame Contribute Delete
400 Bytes
{
"model_max_length": 1024,
"unk_token": "<unk>",
"bos_token": "<s>",
"eos_token": "</s>",
"pad_token": "<pad>",
"additional_special_tokens": [
"<2my>",
"<2en>",
"<formal>",
"<spoken>",
"<bt>",
"<denoise>",
"<mask>"
],
"special_tokens_map": {
"2my": 4,
"2en": 5,
"formal": 6,
"spoken": 7,
"bt": 8,
"denoise": 9,
"mask": 10
}
}