--- license: apache-2.0 base_model: Qwen/Qwen3-VL-4B-Instruct base_model_relation: finetune library_name: transformers pipeline_tag: image-text-to-text tags: - image-captioning - video-captioning - reinforcement-learning - vllm --- # DiffCap-RL 4B DiffCap-RL 4B is a multimodal captioning model fine-tuned from [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) with reinforcement learning. It generates detailed captions for images and videos. This repository contains the full BF16 model weights, tokenizer, chat template, and image/video processors. ## Case Install the dependencies in a fresh environment: ```bash python -m pip install torch==2.8.0 torchvision==0.23.0 transformers==4.57.1 vllm==0.11.0 qwen-vl-utils==0.0.14 Pillow==12.1.0 ``` ```python from pathlib import Path from vllm import LLM, SamplingParams MODEL_ID = "fengyao123/DiffCapRL-4B" # Or the path to downloaded model weights. IMAGE = "https://huggingface.co/datasets/Borise/CaptionQA/resolve/main/natural/images/4.jpg" # To caption your own image, set IMAGE = "/path/to/image.jpg". if __name__ == "__main__": image_url = IMAGE local_media_dir = "" if not IMAGE.startswith(("https://", "http://")): image_path = Path(IMAGE).expanduser().resolve(strict=True) image_url = image_path.as_uri() local_media_dir = str(image_path.parent) llm = LLM( model=MODEL_ID, dtype="bfloat16", tensor_parallel_size=1, gpu_memory_utilization=0.6, max_model_len=32768, max_num_seqs=32, limit_mm_per_prompt={"image": 1, "video": 0}, allowed_local_media_path=local_media_dir, seed=1, generation_config="vllm", enable_chunked_prefill=True, ) sampling = SamplingParams( temperature=0.7, top_p=1.0, top_k=-1, max_tokens=4096, repetition_penalty=1.0, frequency_penalty=0.0, presence_penalty=0.0, ) messages = [ { "role": "user", "content": [ {"type": "text", "text": "Describe this image in detail."}, {"type": "image_url", "image_url": {"url": image_url}}, ], } ] result = llm.chat( messages=[messages], sampling_params=sampling, use_tqdm=False, )[0] prediction = result.outputs[0] print(prediction.text.strip()) if prediction.finish_reason == "length": print("\n[Output limit reached: this caption may be incomplete.]") ``` ## License and limitations Released under Apache-2.0. Retain the base model's license and attribution. Generated captions may contain incorrect or unsupported details; verify them against the input image.