DiffCapRL-4B / README.md
fengyao123's picture
Add files using upload-large-folder tool
a7944a4 verified
|
Raw History Blame Contribute Delete
2.75 kB
---
license: apache-2.0
base_model: Qwen/Qwen3-VL-4B-Instruct
base_model_relation: finetune
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- image-captioning
- video-captioning
- reinforcement-learning
- vllm
---
# DiffCap-RL 4B
DiffCap-RL 4B is a multimodal captioning model fine-tuned from [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) with reinforcement learning. It generates detailed captions for images and videos. This repository contains the full BF16 model weights, tokenizer, chat template, and image/video processors.
## Case
Install the dependencies in a fresh environment:
```bash
python -m pip install torch==2.8.0 torchvision==0.23.0 transformers==4.57.1 vllm==0.11.0 qwen-vl-utils==0.0.14 Pillow==12.1.0
```
```python
from pathlib import Path
from vllm import LLM, SamplingParams
MODEL_ID = "fengyao123/DiffCapRL-4B" # Or the path to downloaded model weights.
IMAGE = "https://huggingface.co/datasets/Borise/CaptionQA/resolve/main/natural/images/4.jpg"
# To caption your own image, set IMAGE = "/path/to/image.jpg".
if __name__ == "__main__":
image_url = IMAGE
local_media_dir = ""
if not IMAGE.startswith(("https://", "http://")):
image_path = Path(IMAGE).expanduser().resolve(strict=True)
image_url = image_path.as_uri()
local_media_dir = str(image_path.parent)
llm = LLM(
model=MODEL_ID,
dtype="bfloat16",
tensor_parallel_size=1,
gpu_memory_utilization=0.6,
max_model_len=32768,
max_num_seqs=32,
limit_mm_per_prompt={"image": 1, "video": 0},
allowed_local_media_path=local_media_dir,
seed=1,
generation_config="vllm",
enable_chunked_prefill=True,
)
sampling = SamplingParams(
temperature=0.7,
top_p=1.0,
top_k=-1,
max_tokens=4096,
repetition_penalty=1.0,
frequency_penalty=0.0,
presence_penalty=0.0,
)
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image in detail."},
{"type": "image_url", "image_url": {"url": image_url}},
],
}
]
result = llm.chat(
messages=[messages],
sampling_params=sampling,
use_tqdm=False,
)[0]
prediction = result.outputs[0]
print(prediction.text.strip())
if prediction.finish_reason == "length":
print("\n[Output limit reached: this caption may be incomplete.]")
```
## License and limitations
Released under Apache-2.0. Retain the base model's license and attribution. Generated captions may contain incorrect or unsupported details; verify them against the input image.