DiffCapRL-4B / README.md
fengyao123's picture
Add files using upload-large-folder tool
a7944a4 verified
|
Raw History Blame Contribute Delete
2.75 kB
metadata
license: apache-2.0
base_model: Qwen/Qwen3-VL-4B-Instruct
base_model_relation: finetune
library_name: transformers
pipeline_tag: image-text-to-text
tags:
  - image-captioning
  - video-captioning
  - reinforcement-learning
  - vllm

DiffCap-RL 4B

DiffCap-RL 4B is a multimodal captioning model fine-tuned from Qwen3-VL-4B-Instruct with reinforcement learning. It generates detailed captions for images and videos. This repository contains the full BF16 model weights, tokenizer, chat template, and image/video processors.

Case

Install the dependencies in a fresh environment:

python -m pip install torch==2.8.0 torchvision==0.23.0 transformers==4.57.1 vllm==0.11.0 qwen-vl-utils==0.0.14 Pillow==12.1.0
from pathlib import Path

from vllm import LLM, SamplingParams

MODEL_ID = "fengyao123/DiffCapRL-4B"  # Or the path to downloaded model weights.
IMAGE = "https://huggingface.co/datasets/Borise/CaptionQA/resolve/main/natural/images/4.jpg"
# To caption your own image, set IMAGE = "/path/to/image.jpg".

if __name__ == "__main__":
    image_url = IMAGE
    local_media_dir = ""
    if not IMAGE.startswith(("https://", "http://")):
        image_path = Path(IMAGE).expanduser().resolve(strict=True)
        image_url = image_path.as_uri()
        local_media_dir = str(image_path.parent)

    llm = LLM(
        model=MODEL_ID,
        dtype="bfloat16",
        tensor_parallel_size=1,
        gpu_memory_utilization=0.6,
        max_model_len=32768,
        max_num_seqs=32,
        limit_mm_per_prompt={"image": 1, "video": 0},
        allowed_local_media_path=local_media_dir,
        seed=1,
        generation_config="vllm",
        enable_chunked_prefill=True,
    )
    sampling = SamplingParams(
        temperature=0.7,
        top_p=1.0,
        top_k=-1,
        max_tokens=4096,
        repetition_penalty=1.0,
        frequency_penalty=0.0,
        presence_penalty=0.0,
    )
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this image in detail."},
                {"type": "image_url", "image_url": {"url": image_url}},
            ],
        }
    ]
    result = llm.chat(
        messages=[messages],
        sampling_params=sampling,
        use_tqdm=False,
    )[0]
    prediction = result.outputs[0]
    print(prediction.text.strip())
    if prediction.finish_reason == "length":
        print("\n[Output limit reached: this caption may be incomplete.]")

License and limitations

Released under Apache-2.0. Retain the base model's license and attribution. Generated captions may contain incorrect or unsupported details; verify them against the input image.