Kimokcheon commited on
Commit
f428126
·
verified ·
1 Parent(s): a21b1cb

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +198 -8
README.md CHANGED
@@ -17,7 +17,7 @@ tags:
17
  arxiv: '2604.08322'
18
  license: mit
19
  base_model:
20
- - Qwen/Qwen2.5-7B-Instruct
21
  ---
22
 
23
  # Fundus-R1
@@ -37,7 +37,7 @@ Fundus-R1 is designed for fundus image understanding, including color fundus pho
37
  | Fundus-R1-3B | `Kimokcheon/Fundus-R1-3B` |
38
  | Fundus-R1-7B | `Kimokcheon/Fundus-R1-7B` |
39
 
40
- This model card is shared by the released Fundus-R1 checkpoints. Please select the checkpoint size according to your compute budget and deployment requirement.
41
 
42
  ## Method Overview
43
 
@@ -75,7 +75,7 @@ import torch
75
  from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
76
  from qwen_vl_utils import process_vision_info
77
 
78
- model_id = "Kimokcheon/Fundus-R1-3B" # or "Kimokcheon/Fundus-R1-7B"
79
 
80
  model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
81
  model_id,
@@ -126,15 +126,205 @@ Install common dependencies:
126
  pip install -U transformers accelerate safetensors qwen-vl-utils
127
  ```
128
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
129
  ## Download Through HF Mirror
130
 
131
- For users in regions where the official Hugging Face endpoint is slow, the checkpoints can be downloaded through the Hugging Face mirror endpoint:
132
 
133
  ```bash
134
  export HF_ENDPOINT=https://hf-mirror.com
135
 
136
- huggingface-cli download Kimokcheon/Fundus-R1-3B --local-dir ./Fundus-R1-3B
137
- huggingface-cli download Kimokcheon/Fundus-R1-7B --local-dir ./Fundus-R1-7B
138
  ```
139
 
140
  To verify mirror availability with a lightweight file:
@@ -142,8 +332,8 @@ To verify mirror availability with a lightweight file:
142
  ```bash
143
  export HF_ENDPOINT=https://hf-mirror.com
144
 
145
- huggingface-cli download Kimokcheon/Fundus-R1-3B config.json --local-dir /tmp/fundus-r1-3b-check
146
- huggingface-cli download Kimokcheon/Fundus-R1-7B config.json --local-dir /tmp/fundus-r1-7b-check
147
  ```
148
 
149
  ## Citation
 
17
  arxiv: '2604.08322'
18
  license: mit
19
  base_model:
20
+ - Qwen/Qwen2.5-VL-7B-Instruct
21
  ---
22
 
23
  # Fundus-R1
 
37
  | Fundus-R1-3B | `Kimokcheon/Fundus-R1-3B` |
38
  | Fundus-R1-7B | `Kimokcheon/Fundus-R1-7B` |
39
 
40
+ The examples below use **Fundus-R1-7B**, based on **Qwen2.5-VL-7B-Instruct**. The 3B checkpoint is listed above as a related model.
41
 
42
  ## Method Overview
43
 
 
75
  from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
76
  from qwen_vl_utils import process_vision_info
77
 
78
+ model_id = "Kimokcheon/Fundus-R1-7B"
79
 
80
  model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
81
  model_id,
 
126
  pip install -U transformers accelerate safetensors qwen-vl-utils
127
  ```
128
 
129
+ ## vLLM Usage
130
+
131
+ Fundus-R1-7B uses the `Qwen2_5_VLForConditionalGeneration` architecture and can be served directly from its Hugging Face checkpoint with vLLM. No GGUF conversion is needed for this route.
132
+
133
+ The following example targets a Linux machine with an NVIDIA GPU, a compatible driver, and sufficient GPU memory. Use a fresh Python environment to avoid conflicts with existing PyTorch installations.
134
+
135
+ ### Start the server
136
+
137
+ ```bash
138
+ python3 -m venv .venv-vllm
139
+ source .venv-vllm/bin/activate
140
+ python -m pip install --upgrade pip
141
+ python -m pip install --upgrade vllm openai
142
+
143
+ CUDA_VISIBLE_DEVICES=0 vllm serve Kimokcheon/Fundus-R1-7B \
144
+ --served-model-name fundus-r1-7b \
145
+ --host 127.0.0.1 \
146
+ --port 8000 \
147
+ --dtype bfloat16 \
148
+ --max-model-len 4096 \
149
+ --max-num-seqs 1 \
150
+ --gpu-memory-utilization 0.90 \
151
+ --limit-mm-per-prompt '{"image":1,"video":0}' \
152
+ --mm-processor-kwargs '{"min_pixels":200704,"max_pixels":1003520}' \
153
+ --enforce-eager
154
+ ```
155
+
156
+ Keep this terminal running. Once the server is ready, check it from another terminal:
157
+
158
+ ```bash
159
+ curl http://127.0.0.1:8000/v1/models
160
+ ```
161
+
162
+ This configuration allows one image per request, limits concurrency to one sequence, and disables CUDA graphs to reduce startup memory pressure. It is a deployment example, **not the paper's evaluation configuration**. Image-resolution limits and generation settings can affect results; increase the image budget when fine visual details are needed and memory permits. These settings do not guarantee that the model will fit on every GPU. For multiple GPUs, adjust `CUDA_VISIBLE_DEVICES` and add the corresponding `--tensor-parallel-size`.
163
+
164
+ The server is bound to the local machine. Do not expose an unauthenticated endpoint containing medical images to a public network.
165
+
166
+ ### Query a local image
167
+
168
+ In a second terminal, activate the same environment and replace `./fundus_image.jpg` with your image path. The image is sent to the local vLLM server as a base64 data URL; no publicly hosted image is required.
169
+
170
+ ```bash
171
+ source .venv-vllm/bin/activate
172
+ python - ./fundus_image.jpg <<'PYTHON'
173
+ import base64
174
+ import mimetypes
175
+ import sys
176
+ from pathlib import Path
177
+ from openai import OpenAI
178
+
179
+ image_path = Path(sys.argv[1]).expanduser()
180
+ if not image_path.is_file():
181
+ raise FileNotFoundError(f"Image not found: {image_path}")
182
+ mime_type = mimetypes.guess_type(image_path.name)[0]
183
+ if mime_type not in {"image/jpeg", "image/png"}:
184
+ raise ValueError("Use a JPEG or PNG image for this example.")
185
+
186
+ image_b64 = base64.b64encode(image_path.read_bytes()).decode("ascii")
187
+ client = OpenAI(
188
+ base_url="http://127.0.0.1:8000/v1",
189
+ api_key="EMPTY", # Local vLLM server; not an OpenAI API key.
190
+ timeout=180.0,
191
+ )
192
+ response = client.chat.completions.create(
193
+ model="fundus-r1-7b",
194
+ messages=[
195
+ {"role": "system", "content": "You are a helpful assistant."},
196
+ {
197
+ "role": "user",
198
+ "content": [
199
+ {"type": "image_url", "image_url": {
200
+ "url": f"data:{mime_type};base64,{image_b64}"
201
+ }},
202
+ {"type": "text", "text":
203
+ "Describe the retinal findings in this fundus image."},
204
+ ],
205
+ },
206
+ ],
207
+ temperature=0.0,
208
+ max_tokens=1024,
209
+ )
210
+ print(response.choices[0].message.content)
211
+ PYTHON
212
+ ```
213
+
214
+ See the [vLLM supported-model list](https://docs.vllm.ai/en/stable/models/supported_models/) and [multimodal input guide](https://docs.vllm.ai/en/stable/features/multimodal_inputs/) for additional deployment options.
215
+
216
+ ## Ollama Usage
217
+
218
+ Use **the Fundus-R1 fine-tuned weights**, not the standard `qwen2.5vl:7b` model from the Ollama library. The latter is the base model, not Fundus-R1.
219
+
220
+ The example below uses a local GGUF conversion and targets recent Ollama releases with support for importing a language-model GGUF together with its multimodal projector. Install or update [Ollama](https://ollama.com/download) first. The commands below use a Bash-compatible shell; keep the Ollama app or service running. If no service is running, launch `ollama serve` in a separate terminal.
221
+
222
+ ### Download and convert the checkpoint
223
+
224
+ Use a separate environment for conversion so its dependencies do not interfere with vLLM.
225
+
226
+ ```bash
227
+ python3 -m venv .venv-ollama
228
+ source .venv-ollama/bin/activate
229
+ python -m pip install --upgrade pip
230
+ python -m pip install --upgrade huggingface_hub ollama
231
+
232
+ hf download Kimokcheon/Fundus-R1-7B --local-dir ./Fundus-R1-7B
233
+
234
+ git clone --depth 1 https://github.com/ggml-org/llama.cpp.git
235
+ python -m pip install -r llama.cpp/requirements/requirements-convert_hf_to_gguf.txt
236
+ mkdir -p ./Fundus-R1-7B-GGUF
237
+
238
+ # Export the fine-tuned language-model weights with Q8_0 quantization.
239
+ python llama.cpp/convert_hf_to_gguf.py ./Fundus-R1-7B \
240
+ --outtype q8_0 \
241
+ --outfile ./Fundus-R1-7B-GGUF/Fundus-R1-7B-Q8_0.gguf
242
+
243
+ # Export the matching vision encoder/projector from the SAME checkpoint.
244
+ python llama.cpp/convert_hf_to_gguf.py ./Fundus-R1-7B \
245
+ --mmproj \
246
+ --outtype f16 \
247
+ --outfile ./Fundus-R1-7B-GGUF/mmproj-Fundus-R1-7B-f16.gguf
248
+ ```
249
+
250
+ Both files are required for image understanding. Do not substitute a projector from another checkpoint: the fine-tuned visual weights must be preserved. Keep the original Hugging Face checkpoint for full-precision inference and evaluation. Q8_0 conversion changes the weight precision and can change model outputs.
251
+
252
+ ### Create the Ollama model
253
+
254
+ Run the following from the directory containing `Fundus-R1-7B-GGUF`:
255
+
256
+ ```bash
257
+ cat > Modelfile <<'MODELFILE'
258
+ FROM ./Fundus-R1-7B-GGUF/Fundus-R1-7B-Q8_0.gguf
259
+ FROM ./Fundus-R1-7B-GGUF/mmproj-Fundus-R1-7B-f16.gguf
260
+
261
+ SYSTEM """You are a helpful assistant."""
262
+ TEMPLATE """{{- range .Messages }}
263
+ {{- printf "<|im_start|>%s\n%s<|im_end|>\n" .Role .Content }}
264
+ {{- end }}
265
+ {{- printf "<|im_start|>assistant\n" }}"""
266
+
267
+ PARAMETER num_ctx 4096
268
+ PARAMETER num_predict 1024
269
+ PARAMETER temperature 0
270
+ PARAMETER stop "<|im_end|>"
271
+ PARAMETER stop "<|endoftext|>"
272
+ MODELFILE
273
+
274
+ ollama create fundus-r1-7b -f ./Modelfile
275
+ ollama show fundus-r1-7b
276
+ ```
277
+
278
+ Check that `ollama show` lists the `vision` capability before querying images. If your version rejects the two-file import or shows only text capabilities, update Ollama and check the import logs; use the vLLM route above rather than dropping the projector.
279
+
280
+ ### Query a local image
281
+
282
+ ```bash
283
+ source .venv-ollama/bin/activate
284
+ python - ./fundus_image.jpg <<'PYTHON'
285
+ import sys
286
+ from pathlib import Path
287
+ from ollama import chat
288
+
289
+ image_path = Path(sys.argv[1]).expanduser().resolve()
290
+ if not image_path.is_file():
291
+ raise FileNotFoundError(f"Image not found: {image_path}")
292
+
293
+ response = chat(
294
+ model="fundus-r1-7b",
295
+ messages=[
296
+ {"role": "system", "content": "You are a helpful assistant."},
297
+ {
298
+ "role": "user",
299
+ "content": "Describe the retinal findings in this fundus image.",
300
+ "images": [str(image_path)],
301
+ },
302
+ ],
303
+ options={"temperature": 0.0, "num_predict": 1024},
304
+ )
305
+ print(response.message.content)
306
+ PYTHON
307
+ ```
308
+
309
+ Alternatively, provide an image path directly in the Ollama CLI:
310
+
311
+ ```bash
312
+ ollama run fundus-r1-7b ./fundus_image.jpg "Describe the retinal findings in this fundus image."
313
+ ```
314
+
315
+ See the [Ollama Modelfile reference](https://docs.ollama.com/modelfile), [Ollama vision guide](https://docs.ollama.com/capabilities/vision), and [llama.cpp multimodal conversion guide](https://github.com/ggml-org/llama.cpp/blob/master/tools/mtmd/README.md).
316
+
317
+ **Validation status:** These deployment recipes are based on the checkpoint architecture and upstream documentation/code. They have not been end-to-end tested with this checkpoint. Validate image-conditioned outputs in your environment before relying on either integration; do not assume converted or quantized outputs reproduce the paper's reported results. The medical-use disclaimer above applies to both backends.
318
+
319
  ## Download Through HF Mirror
320
 
321
+ For users in regions where the official Hugging Face endpoint is slow, the checkpoints can be downloaded through the Hugging Face mirror endpoint. Install the CLI with `python -m pip install --upgrade huggingface_hub` if it is not already available:
322
 
323
  ```bash
324
  export HF_ENDPOINT=https://hf-mirror.com
325
 
326
+ hf download Kimokcheon/Fundus-R1-3B --local-dir ./Fundus-R1-3B
327
+ hf download Kimokcheon/Fundus-R1-7B --local-dir ./Fundus-R1-7B
328
  ```
329
 
330
  To verify mirror availability with a lightweight file:
 
332
  ```bash
333
  export HF_ENDPOINT=https://hf-mirror.com
334
 
335
+ hf download Kimokcheon/Fundus-R1-3B config.json --local-dir /tmp/fundus-r1-3b-check
336
+ hf download Kimokcheon/Fundus-R1-7B config.json --local-dir /tmp/fundus-r1-7b-check
337
  ```
338
 
339
  ## Citation