Image-Text-to-Text
Transformers
Safetensors
llava_onevision2_5
text-generation
multimodal
video-understanding
streaming
llava-onevision
conversational
custom_code
Instructions to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", trust_remote_code=True) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct
- SGLang
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with Docker Model Runner:
docker model run hf.co/lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct
TimeLens ActivityNet: Qwen score at the aligned setting
Browse files- README.md +2 -2
- README_CN.md +2 -2
README.md
CHANGED
|
@@ -115,7 +115,7 @@ OV2.5-4B (stage 4, iteration 11750) against **Qwen3.5-4B with thinking disabled*
|
|
| 115 |
| LongVideoBench val | 62.68 | 700f × 300,000 | 68.96 | 1280f × 80,000 | 71.06 | TC768 × 235,200 |
|
| 116 |
| VideoEval-Pro | 62.84 | 2048f × 100,000 | 64.39 | 950f × 200,000 | 64.62 | TC768 × 235,200 |
|
| 117 |
| **Temporal grounding (mIoU)** | | | | | | |
|
| 118 |
-
| TimeLens ActivityNet |
|
| 119 |
| TimeLens Charades | 48.05 | 2048f × 409,600, fps=4 | 60.58 | 2048f × 409,600, fps=4 | 61.25 | TC128 × 313,600 |
|
| 120 |
| TimeLens QVHighlights | 67.51 | 2048f × 409,600, fps=4 | 68.39 | 2048f × 409,600, fps=4 | 70.31 | TC128 × 313,600 |
|
| 121 |
|
|
@@ -124,7 +124,7 @@ OV2.5-4B (stage 4, iteration 11750) against **Qwen3.5-4B with thinking disabled*
|
|
| 124 |
\* Video-Index 512f: frames are supplied by the task as images, so both sides take multi-image input. The codec number comes from the official-budget run (TC256 cap); the matching frame-sampling control on the video files scores 19.05.
|
| 125 |
\* EgoSchema: the 500-question public subset. Answers to the full 5031 questions are not public.
|
| 126 |
\* VideoMME with subtitles and MMOU: Qwen has no comparable result on the official path. Qwen's VideoMME long with subtitles run was invalid (judge recorded 0/900) and is left out rather than reported.
|
| 127 |
-
\* Settings are the ones that produced each reported score. Most rows are matched
|
| 128 |
\* Where a score is the best over several settings, the search was uneven and mostly favoured Qwen: VideoMME long no-subtitles 28 settings vs our 6, LVBench 27 vs 16, LongVideoBench 28 vs 2, VideoEval-Pro 28 vs 2.
|
| 129 |
\* The codec column is scored by rule, since the answers are option letters and the judge is equivalent there.
|
| 130 |
|
|
|
|
| 115 |
| LongVideoBench val | 62.68 | 700f × 300,000 | 68.96 | 1280f × 80,000 | 71.06 | TC768 × 235,200 |
|
| 116 |
| VideoEval-Pro | 62.84 | 2048f × 100,000 | 64.39 | 950f × 200,000 | 64.62 | TC768 × 235,200 |
|
| 117 |
| **Temporal grounding (mIoU)** | | | | | | |
|
| 118 |
+
| TimeLens ActivityNet | 53.84 | 2048f × 409,600, fps=4 | 58.84 | 2048f × 409,600, fps=4 | 59.11 | TC128 × 313,600 |
|
| 119 |
| TimeLens Charades | 48.05 | 2048f × 409,600, fps=4 | 60.58 | 2048f × 409,600, fps=4 | 61.25 | TC128 × 313,600 |
|
| 120 |
| TimeLens QVHighlights | 67.51 | 2048f × 409,600, fps=4 | 68.39 | 2048f × 409,600, fps=4 | 70.31 | TC128 × 313,600 |
|
| 121 |
|
|
|
|
| 124 |
\* Video-Index 512f: frames are supplied by the task as images, so both sides take multi-image input. The codec number comes from the official-budget run (TC256 cap); the matching frame-sampling control on the video files scores 19.05.
|
| 125 |
\* EgoSchema: the 500-question public subset. Answers to the full 5031 questions are not public.
|
| 126 |
\* VideoMME with subtitles and MMOU: Qwen has no comparable result on the official path. Qwen's VideoMME long with subtitles run was invalid (judge recorded 0/900) and is left out rather than reported.
|
| 127 |
+
\* Settings are the ones that produced each reported score. Most rows are matched; two are not: on **TOMATO** Qwen ran 16 frames at fps=2 while we used the model defaults, and on **MMOU** only we have a score.
|
| 128 |
\* Where a score is the best over several settings, the search was uneven and mostly favoured Qwen: VideoMME long no-subtitles 28 settings vs our 6, LVBench 27 vs 16, LongVideoBench 28 vs 2, VideoEval-Pro 28 vs 2.
|
| 129 |
\* The codec column is scored by rule, since the answers are option letters and the judge is equivalent there.
|
| 130 |
|
README_CN.md
CHANGED
|
@@ -115,7 +115,7 @@ OV2.5-4B(stage 4,iteration 11750)对 **Qwen3.5-4B(关闭 thinking)**
|
|
| 115 |
| LongVideoBench val | 62.68 | 700f × 300,000 | 68.96 | 1280f × 80,000 | 71.06 | TC768 × 235,200 |
|
| 116 |
| VideoEval-Pro | 62.84 | 2048f × 100,000 | 64.39 | 950f × 200,000 | 64.62 | TC768 × 235,200 |
|
| 117 |
| **时间定位(mIoU)** | | | | | | |
|
| 118 |
-
| TimeLens ActivityNet |
|
| 119 |
| TimeLens Charades | 48.05 | 2048f × 409,600, fps=4 | 60.58 | 2048f × 409,600, fps=4 | 61.25 | TC128 × 313,600 |
|
| 120 |
| TimeLens QVHighlights | 67.51 | 2048f × 409,600, fps=4 | 68.39 | 2048f × 409,600, fps=4 | 70.31 | TC128 × 313,600 |
|
| 121 |
|
|
@@ -124,7 +124,7 @@ OV2.5-4B(stage 4,iteration 11750)对 **Qwen3.5-4B(关闭 thinking)**
|
|
| 124 |
\* Video-Index 512f:帧由任务以图片给出,两边都是多图输入。codec 一列取自官方预算测试(TC256 上限);同一批的抽帧对照(视频文件输入)为 19.05。
|
| 125 |
\* EgoSchema:公开答案的 500 题子集。全量 5031 题答案不公开。
|
| 126 |
\* VideoMME 有字幕三项与 MMOU:Qwen 在官方路径下没有可比结果。Qwen 的 VideoMME long 有字幕那次运行无效(judge 记录为 0/900),故留空而非报出。
|
| 127 |
-
\* 设置列是各自取得该分数时的实际设置。多数行两边一致,
|
| 128 |
\* 对「取多设置最高分」的行,两边搜索量并不对等,且多数对 Qwen 更有利:VideoMME long 无字幕 28 个设置对我方 6 个,LVBench 27 对 16,LongVideoBench 28 对 2,VideoEval-Pro 28 对 2。
|
| 129 |
\* codec 一列为规则分——回答均为选项字母,与 judge 等价。
|
| 130 |
|
|
|
|
| 115 |
| LongVideoBench val | 62.68 | 700f × 300,000 | 68.96 | 1280f × 80,000 | 71.06 | TC768 × 235,200 |
|
| 116 |
| VideoEval-Pro | 62.84 | 2048f × 100,000 | 64.39 | 950f × 200,000 | 64.62 | TC768 × 235,200 |
|
| 117 |
| **时间定位(mIoU)** | | | | | | |
|
| 118 |
+
| TimeLens ActivityNet | 53.84 | 2048f × 409,600, fps=4 | 58.84 | 2048f × 409,600, fps=4 | 59.11 | TC128 × 313,600 |
|
| 119 |
| TimeLens Charades | 48.05 | 2048f × 409,600, fps=4 | 60.58 | 2048f × 409,600, fps=4 | 61.25 | TC128 × 313,600 |
|
| 120 |
| TimeLens QVHighlights | 67.51 | 2048f × 409,600, fps=4 | 68.39 | 2048f × 409,600, fps=4 | 70.31 | TC128 × 313,600 |
|
| 121 |
|
|
|
|
| 124 |
\* Video-Index 512f:帧由任务以图片给出,两边都是多图输入。codec 一列取自官方预算测试(TC256 上限);同一批的抽帧对照(视频文件输入)为 19.05。
|
| 125 |
\* EgoSchema:公开答案的 500 题子集。全量 5031 题答案不公开。
|
| 126 |
\* VideoMME 有字幕三项与 MMOU:Qwen 在官方路径下没有可比结果。Qwen 的 VideoMME long 有字幕那次运行无效(judge 记录为 0/900),故留空而非报出。
|
| 127 |
+
\* 设置列是各自取得该分数时的实际设置。多数行两边一致,有两行不是:**TOMATO** Qwen 为 16 帧 fps=2 而我方用模型默认值;**MMOU** 只有我方有分。
|
| 128 |
\* 对「取多设置最高分」的行,两边搜索量并不对等,且多数对 Qwen 更有利:VideoMME long 无字幕 28 个设置对我方 6 个,LVBench 27 对 16,LongVideoBench 28 对 2,VideoEval-Pro 28 对 2。
|
| 129 |
\* codec 一列为规则分——回答均为选项字母,与 judge 等价。
|
| 130 |
|