Stop generation at <|im_end|> (add generation_config.json)

#1

The repo has no generation_config.json, so the end-of-sequence token comes from the tokenizer config: the base model's <|endoftext|> (248044). The model was trained to end its turns with <|im_end|> (248046), which is what the Tinker renderer stops on. Inference stacks therefore don't stop at the end of a turn: they skip the special token and the model goes on to write <|im_start|>assistant and another turn. At max_new_tokens=160 this is mostly hidden by the cap.

This adds a generation_config.json with eos_token_id: [248046, 248044], which both transformers generate() and vLLM pick up.

Tested on vLLM 0.18 serving this repo, 8 samples at max_tokens=512 with no client-side stop tokens:

  • before: 7/8 ran to the length limit and leaked a fake next turn (...How do you view jazz?<|im_end|>\n<|im_start|>assistant\nI've done some with code...)
  • after: 8/8 finished with stop, 0 leaked

(Thanks for releasing all this!)

caseyeasd changed pull request status to open
caseyeasd changed pull request status to merged

Sign up or log in to comment