README: stop at <|im_end|> when serving

#1

Same issue as in caseyeasd/claudefriend: Qwen/Qwen3.5-9B-Base's config ends generation at <|endoftext|> (248044), but this model ends its turns with <|im_end|> (248046). Served as in the README, it runs past the end of its turn and writes the next turn itself. A LoRA repo can't change the base model's generation config, so this adds a note to the README: pass stop_token_ids=[248046] to vLLM, or eos_token_id=[248046, 248044] to generate().

I checked stop_token_ids=[248046] on vLLM 0.18 with the merged model: replies end cleanly at the end of the turn.

caseyeasd changed pull request status to open
caseyeasd changed pull request status to merged

Sign up or log in to comment