How to use from
Docker Model Runner
docker model run hf.co/fengyao123/DiffCapRL-4B
Quick Links

DiffCap-RL 4B

DiffCap-RL 4B is a multimodal captioning model fine-tuned from Qwen3-VL-4B-Instruct with reinforcement learning. It generates detailed captions for images and videos. This repository contains the full BF16 model weights, tokenizer, chat template, and image/video processors.

Case

Install the dependencies in a fresh environment:

python -m pip install torch==2.8.0 torchvision==0.23.0 transformers==4.57.1 vllm==0.11.0 qwen-vl-utils==0.0.14 Pillow==12.1.0
from pathlib import Path

from vllm import LLM, SamplingParams

MODEL_ID = "fengyao123/DiffCapRL-4B"  # Or the path to downloaded model weights.
IMAGE = "https://huggingface.co/datasets/Borise/CaptionQA/resolve/main/natural/images/4.jpg"
# To caption your own image, set IMAGE = "/path/to/image.jpg".

if __name__ == "__main__":
    image_url = IMAGE
    local_media_dir = ""
    if not IMAGE.startswith(("https://", "http://")):
        image_path = Path(IMAGE).expanduser().resolve(strict=True)
        image_url = image_path.as_uri()
        local_media_dir = str(image_path.parent)

    llm = LLM(
        model=MODEL_ID,
        dtype="bfloat16",
        tensor_parallel_size=1,
        gpu_memory_utilization=0.6,
        max_model_len=32768,
        max_num_seqs=32,
        limit_mm_per_prompt={"image": 1, "video": 0},
        allowed_local_media_path=local_media_dir,
        seed=1,
        generation_config="vllm",
        enable_chunked_prefill=True,
    )
    sampling = SamplingParams(
        temperature=0.7,
        top_p=1.0,
        top_k=-1,
        max_tokens=4096,
        repetition_penalty=1.0,
        frequency_penalty=0.0,
        presence_penalty=0.0,
    )
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this image in detail."},
                {"type": "image_url", "image_url": {"url": image_url}},
            ],
        }
    ]
    result = llm.chat(
        messages=[messages],
        sampling_params=sampling,
        use_tqdm=False,
    )[0]
    prediction = result.outputs[0]
    print(prediction.text.strip())
    if prediction.finish_reason == "length":
        print("\n[Output limit reached: this caption may be incomplete.]")

License and limitations

Released under Apache-2.0. Retain the base model's license and attribution. Generated captions may contain incorrect or unsupported details; verify them against the input image.

Downloads last month
25
Safetensors
Model size
5B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for fengyao123/DiffCapRL-4B

Finetuned
(566)
this model