README: stop at <|im_end|> when serving

#1
Files changed (1) hide show
  1. README.md +8 -0
README.md CHANGED
@@ -27,6 +27,13 @@ vllm serve Qwen/Qwen3.5-9B-Base --enable-lora --max-lora-rank 32 \
27
  --lora-modules claudefriend=caseyeasd/claudefriend-lora
28
  ```
29
 
 
 
 
 
 
 
 
30
  With PEFT:
31
 
32
  ```python
@@ -37,4 +44,5 @@ from transformers import AutoModelForImageTextToText, AutoTokenizer
37
  tok = AutoTokenizer.from_pretrained("caseyeasd/claudefriend")
38
  base = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.5-9B-Base", dtype=torch.bfloat16)
39
  model = PeftModel.from_pretrained(base, "caseyeasd/claudefriend-lora")
 
40
  ```
 
27
  --lora-modules claudefriend=caseyeasd/claudefriend-lora
28
  ```
29
 
30
+ The base model's config ends generation at `<|endoftext|>`, but the model ends its turns with `<|im_end|>` (id 248046), so pass that as a stop token on every request, or it will keep going and write the next turn itself:
31
+
32
+ ```python
33
+ client.chat.completions.create(model="claudefriend", messages=messages, max_tokens=160,
34
+ extra_body={"stop_token_ids": [248046]})
35
+ ```
36
+
37
  With PEFT:
38
 
39
  ```python
 
44
  tok = AutoTokenizer.from_pretrained("caseyeasd/claudefriend")
45
  base = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.5-9B-Base", dtype=torch.bfloat16)
46
  model = PeftModel.from_pretrained(base, "caseyeasd/claudefriend-lora")
47
+ # when generating, stop at the end of the turn: model.generate(..., eos_token_id=[248046, 248044])
48
  ```