Add added_tokens_decoder to tokenizer_config.json

#4
Reson8 org

Lists the 68 added/special tokens (already in tokenizer.json) in tokenizer_config.json.

Transformers output is unchanged (verified: identical input_ids, vocab and special ids). Runtimes that read only tokenizer_config.json, such as the mlx-qwen3-asr MLX port, otherwise fall back to the stock Qwen3 ids (audio_pad 151676 instead of 99595) and fail.

MarktHart changed pull request status to merged

Sign up or log in to comment