Instructions to use caseyeasd/claudefriend-lora with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use caseyeasd/claudefriend-lora with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-9B-Base") model = PeftModel.from_pretrained(base_model, "caseyeasd/claudefriend-lora") - Notebooks
- Google Colab
- Kaggle
README: stop at <|im_end|> when serving
#1
by maxsloef - opened
README.md
CHANGED
|
@@ -27,6 +27,13 @@ vllm serve Qwen/Qwen3.5-9B-Base --enable-lora --max-lora-rank 32 \
|
|
| 27 |
--lora-modules claudefriend=caseyeasd/claudefriend-lora
|
| 28 |
```
|
| 29 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
With PEFT:
|
| 31 |
|
| 32 |
```python
|
|
@@ -37,4 +44,5 @@ from transformers import AutoModelForImageTextToText, AutoTokenizer
|
|
| 37 |
tok = AutoTokenizer.from_pretrained("caseyeasd/claudefriend")
|
| 38 |
base = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.5-9B-Base", dtype=torch.bfloat16)
|
| 39 |
model = PeftModel.from_pretrained(base, "caseyeasd/claudefriend-lora")
|
|
|
|
| 40 |
```
|
|
|
|
| 27 |
--lora-modules claudefriend=caseyeasd/claudefriend-lora
|
| 28 |
```
|
| 29 |
|
| 30 |
+
The base model's config ends generation at `<|endoftext|>`, but the model ends its turns with `<|im_end|>` (id 248046), so pass that as a stop token on every request, or it will keep going and write the next turn itself:
|
| 31 |
+
|
| 32 |
+
```python
|
| 33 |
+
client.chat.completions.create(model="claudefriend", messages=messages, max_tokens=160,
|
| 34 |
+
extra_body={"stop_token_ids": [248046]})
|
| 35 |
+
```
|
| 36 |
+
|
| 37 |
With PEFT:
|
| 38 |
|
| 39 |
```python
|
|
|
|
| 44 |
tok = AutoTokenizer.from_pretrained("caseyeasd/claudefriend")
|
| 45 |
base = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.5-9B-Base", dtype=torch.bfloat16)
|
| 46 |
model = PeftModel.from_pretrained(base, "caseyeasd/claudefriend-lora")
|
| 47 |
+
# when generating, stop at the end of the turn: model.generate(..., eos_token_id=[248046, 248044])
|
| 48 |
```
|