samog0d commited on
Commit
8252560
·
verified ·
1 Parent(s): 9b7e3a2

fix vLLM instructions for adapter

Browse files
Files changed (1) hide show
  1. README.md +14 -3
README.md CHANGED
@@ -93,6 +93,10 @@ incident response chain. Keep an analyst in the loop.
93
 
94
  ## Getting Started
95
 
 
 
 
 
96
  ### Transformers + Unsloth
97
 
98
  ```python
@@ -119,21 +123,28 @@ print(ask("Analyze the following CVE description and map it to the appropriate C
119
 
120
  ### vLLM
121
 
122
- Merge the adapter into the base, then serve text-only:
123
 
124
  ```bash
125
- vllm serve BeyondMemory/BeyondCTI-27B \
 
 
 
126
  --language-model-only \
127
  --reasoning-parser qwen3 \
128
  --max-model-len 16384
129
  ```
130
 
 
 
 
 
131
  ```python
132
  from openai import OpenAI
133
  client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
134
 
135
  r = client.chat.completions.create(
136
- model="BeyondMemory/BeyondCTI-27B", temperature=0,
137
  messages=[
138
  {"role": "system", "content": "You are a cybersecurity expert specializing in cyberthreat intelligence."},
139
  {"role": "user", "content": "Analyze the following CVE description and calculate the CVSS v3.1 "
 
93
 
94
  ## Getting Started
95
 
96
+ This repository holds the **LoRA adapter** (~640 MB), not merged weights. The base model
97
+ `unsloth/Qwen3.6-27B` is downloaded separately and the adapter is applied on top, so both loaders below
98
+ pull roughly 55 GB on first run.
99
+
100
  ### Transformers + Unsloth
101
 
102
  ```python
 
123
 
124
  ### vLLM
125
 
126
+ Serve the base text-only and attach the adapter as a LoRA module:
127
 
128
  ```bash
129
+ vllm serve unsloth/Qwen3.6-27B \
130
+ --enable-lora \
131
+ --lora-modules beyondcti=BeyondMemory/BeyondCTI-27B \
132
+ --max-lora-rank 32 \
133
  --language-model-only \
134
  --reasoning-parser qwen3 \
135
  --max-model-len 16384
136
  ```
137
 
138
+ Then call it with `model="beyondcti"`. If you would rather serve a single merged checkpoint, merge
139
+ locally first with `PeftModel.from_pretrained(base, adapter).merge_and_unload()` and point vLLM at the
140
+ output directory.
141
+
142
  ```python
143
  from openai import OpenAI
144
  client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
145
 
146
  r = client.chat.completions.create(
147
+ model="beyondcti", temperature=0,
148
  messages=[
149
  {"role": "system", "content": "You are a cybersecurity expert specializing in cyberthreat intelligence."},
150
  {"role": "user", "content": "Analyze the following CVE description and calculate the CVSS v3.1 "