generation_config.json stops on <EOS_TOKEN> (id 3), but chat turns end with <|END_OF_TURN_TOKEN|> (id 6)
Hi, thanks for releasing Tiny Aya. I found a small mismatch in the stop token settings that wastes generation time.
What I see
generation_config.json and config.json set eos_token_id to 3 (<EOS_TOKEN>), but the tokenizer's EOS token is <|END_OF_TURN_TOKEN|> (id 6), which is what ends an assistant turn in the chat template.
from transformers import AutoTokenizer, GenerationConfig
m = "CohereLabs/tiny-aya-global"
print(GenerationConfig.from_pretrained(m).eos_token_id) # 3
print(AutoTokenizer.from_pretrained(m).eos_token_id) # 6
Checked on revision af89d219.
Effect
With model.generate(...) and the default generation config, generation only stops on token 3. In my runs with greedy decoding, the model finished its answer with token 6 and then kept emitting token 6 until max_new_tokens. The extra tokens are special tokens, so they disappear when decoding with skip_special_tokens=True and the text looks normal. The cost is hidden: every answer uses the full token budget, and a library that only returns decoded text cannot tell that the answer finished early.
After passing both ids as stop tokens, one batch of four answers went from about 30 s to about 7 s on my machine (768 new tokens allowed).
Workaround
model.generate(**inputs, eos_token_id=[3, 6], ...)
Suggested fix
Set "eos_token_id": [3, 6] in generation_config.json, so chat generation stops at the end of the turn by default. It may be worth checking the other Tiny Aya variants for the same setting.